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EXECUTIVE  SUMMARY 


PROBLEM 

The  Single  Integrated  Air  Picture  (SIAP)  System  Engineering  Task  Force  (SE  TF) 
requires  a  single,  coherent  analytic  approach  that  delineates  overarching  SIAP 
related  analysis  objectives.  Such  an  approach  will  also  serve  as  a  roadmap  for 
the  integration  of  selected  analysis  endeavors.  The  System  Engineer  (SE)  will 
use  the  Integrated  Assessment  Plan  (IAP)  to  guide  and  facilitate  the  comparison 
of  results  across  different  assessment  venues  and  to  leverage  each  venue’s 
strengths.  Such  an  integrated  approach  requires  standardized  collaborative 
teams,  tools,  and  processes  that  support  focused,  repeatable,  and  rigorous 
analysis  of  the  SIAP.  This  IAP  describes  and  articulates  the  multiple  analytical 
venues  available  to  the  SIAP  and  provides  a  focused  implementation  plan 
documenting  the  strategy  for  integration  of  multiple  SIAP  analysis  efforts  across 
multiple  SIAP  venues.  This  plan  documents  a  coherent  process  for  the 
integration  of  the  results  from  the  many  different  SIAP  analytical  endeavors. 

OBJECTIVES 

Define  the  overarching  SIAP  related  analytical  methodologies,  and  analysis 
objectives,  as  applicable  for  particular  SIAP  block  analytical  efforts.  Lay  out  a 
roadmap  for  integrating  the  individual  SIAP  related  analysis  results  across 
multiple  analytical  venues.  Doing  so  will  enable  the  SE  to  consistently  and 
credibly  assess  Integrated  Air  Defense  System  (IADS)  performance  and  make 
legitimate  and  justifiable  recommendations  to  the  Joint  Requirements  Oversight 
Council  (JROC). 

APPROACH 

Describe  the  different  analyses  and  analytical  venues  sponsored,  supported,  or 
leveraged  by  the  SIAP  SE.  Describe  standard  teams,  tools,  and  processes  used 
by  the  SIAP  SE.  Structure  the  IAP  to  provide  general  concepts  and  guidance 
that  apply  to  all  SIAP  analysis  in  a  main  document,  with  the  details  of  each  block 
analysis  strategy  defined  in  separate  block  appendices. 

CONCLUSIONS 

To  ensure  proper  synergy  and  leveraging  across  a  host  of  analytical  venues,  a 
single  assessment  plan  must  account  for  options,  categorize  the  strengths  and 
limitations  of  each  venue,  and  describe  a  single,  coherent  plan  for  conducting 
thorough  and  sound  block  analyses.  Enacting  and  executing  the  enclosed  IAP  is 
the  best  guarantee  of  establishing  and  maintaining  a  rigorous  and  disciplined 
analytical  process  for  the  SIAP  SE. 

RECOMMENDATIONS 

The  IAP  shall  be  used  as  the  primary  planning  and  execution  document  for  block 
system  engineering  analyses  for  the  SIAP  SE. 
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1.  Introduction 

This  plan  documents  a  coherent  analytical  approach  that  delineates  the 
overarching  analysis  objectives,  approach,  and  processes  for  Single  Integrated 
Air  Picture  (SIAP)  related  analysis.  There  is  no  existing,  explicit  and 
comprehensive  process  for  assessing  the  Integrated  Air  Defense  System  (IADS) 
Joint  Data  Network  (JDN)  performance.  As  a  result,  most  of  the  Department  of 
Defense’s  (DoD)  efforts  to  date  to  correct  systemic,  doctrinal,  and  process  issues 
contributing  to  SIAP  shortfalls  have  proven  ineffective  or  incomplete.  These 
shortfalls  result  in,  among  other  things,  reduced  system  capability  and  difficulty  or 
inability  to  anticipate  and  address  emerging  SIAP  deficiencies.  The  SIAP 
System  Engineer  (SE)  developed  the  Integrated  Assessment  Plan  (IAP)  to  help 
mitigate  this  problem.  This  IAP  provides  an  overarching  strategy  and 
implementation  approach  for  conducting  SIAP  related  analysis  and  IADS 
performance  assessment. 

The  SIAP  SE  is  responsible  for  recommending  SIAP  improvements  to  the 
Joint  Requirements  Oversight  Council  (JROC)  based  on  results  from  a 
disciplined  system  engineering  process.  A  rigorous  set  of  analyses,  as  part  of  a 
disciplined  system  engineering  process,  is  a  key  enabler  to  providing  said 
recommendations.  This  IAP  lays  out  the  fundamental  steps  inherent  in  applying 
rigorous  system  analysis  to  assessment  of  IADS  performance.  It  promotes  joint 
collaboration  and  focus,  and  allows  many  types  of  analysis  to  be  conducted  for 
one  central  analytical  body,  the  SIAP  Analysis  Team  (SAT).  The  SAT  is  a 
collaborative  (among  the  Services  and  Agencies  under  the  leadership  of  the 
SIAP  SE)  engineering  and  analysis  team.  This  method  adds  focus,  order,  and 
value  to  the  many  SIAP  related  analyses  being  conducted.  Effective  execution  of 
the  IAP  will  result  in  timely  and  meaningful  recommendations  to  the  JROC. 

The  system  engineering  process  will  consider  the  many  Service/Agency 
(S/A)  analysis  practices  already  in  existence  to  provide  the  best  answer  possible 
for  various  aspects  of  analysis  issues  and  goals.  Currently  there  exist  at  least 
four  distinct  venues  that  are  used  by  analysis  agencies.  For  the  purpose  of  this 
document  and  SIAP  related  activities,  a  venue  is  considered  to  be  an 
assessment  structure  available  to  the  SIAP  SE.  The  four  venues  addressed  in 
this  IAP  are  live  exercise,  hardware-in-the-loop  (HWIL),  operator-in-the-loop 
(OITL),  and  constructive  simulations.  Within  these  venues  there  can  literally  be 
hundreds  of  analysis  tools  available.  For  the  purposes  of  this  IAP,  the  SIAP  SE 
defines  tools  as  sub-items  of  venues.  Examples  include  Joint  Combat 
Identification  Evaluation  Team  (JCIET)  (a  live  exercise  tool),  Joint  Distributed 
Engineering  Plant  (JDEP)  (a  HWIL  tool),  Virtual  Warfare  Center  (VWC)  (an  OITL 
tool),  and  Extended  Air  Defense  Test  Bed  (EADTB)  (a  constructive  simulation). 
Additionally,  a  one-time  execution  of  a  particular  tool  is  referred  to  as  an  event. 

To  date,  there  has  not  been  an  attempt  to  coordinate  or  synergize  across  the 
many  Joint  potential  assessment  venues  and  related  tools  available.  The  result 
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is  that  while  many  venues  and  related  tools  characterize  some  aspect  of  SIAP 
related  performance  in  similar  ways,  none  are  standardized  enough  to  equitably 
compare  results  from  one  venue  with  another,  or  to  build  a  consistent  cumulative 
story  of  SIAP  performance  based  on  their  multiple  contributions.  In  short,  there 
is  no  current  systematic  and  detailed  IADS  performance  assessment 
implementation  plan  available  to  the  SIAP  SE. 

This  document  describes  the  different  types  of  analyses  to  be  conducted  and 
leveraged  by  the  SIAP  SE  in  adherence  to  a  disciplined  system  engineering 
process.  It  also  describes  the  standard  teams,  tools,  and  processes  that  support 
the  analyses. 


2.  Analysis  Requirements 


The  SIAP  Systems  Engineering  Task  Force  (SE  TF)  Charter  mandated  the 
implementation  of  a  “disciplined  system  engineering  process”  to  “achieve  a  SIAP 
that  satisfies  the  warfighter  needs.”  The  SIAP  SE’s  system  engineering  process 
will  draw  upon  the  IEEE  STD  1220-1998  process  and  be  further  articulated  in  the 
SIAP  System  Engineering  Management  Plan  (SEMP).  This  process  entails 
effective  root  cause  analysis  (RCA).  The  results  of  root  cause  analysis  will 
support  the  subsequent  development  of  proposed  solutions  and 
recommendations.  Performance  and  cost  benefit  analyses  will  address 
questions  “So  What?”  and  “How  much?”  respectively.  The  overall  SIAP  SE 
analysis  plan  for  achieving  the  objective  SIAP  is  represented  below  in  Figure  1 . 


INCREMENTAL 
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OF  2010  JTAMD 
INTEGRATED 
ARCHITECTURE 


S I A  P  QB  J  ECTI VE 
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Engineering 
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Analysis 
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Figure  1 .  SIAP  SE  Analysis  Plan 
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The  SIAP  SE  will  recommend  improvements  to  the  IADS  in  incremental 
“block”  upgrades.  Incremental  block  upgrades  will  begin  with  near  term  JDN 
fixes  that  improve  the  current  air  picture  and  specifically  satisfy  the  requirements 
defined  by  capstone  requirements  documents.  By  consolidating  JDN 
improvements  into  logical  block  upgrades,  and  by  coordinating  these  upgrades 
with  other  planned  changes  to  host  systems,  the  SIAP  SE  minimizes  the  number 
of  times  that  host  computer  programs  must  be  modified  and  tested  and  thereby 
minimizes  cost.  The  initial  block  upgrades  will  target  Link  16  deficiencies  of  the 
JDN. 

For  each  block,  the  SIAP  SE  will  establish  Block  Working  Groups  to  conduct 
engineering  level  analysis  of  issues  related  to  that  block.  In  accordance  with  the 
SEMP,  Block  Working  Groups  will  functionally  decompose  problems  uncovered 
during  root  cause  analysis  and  develop  solutions  to  block  issues  by  modifying  the 
relevant  functional  processes.  Before  recommending  solutions  for  theater  level 
analysis,  working  groups  will  determine  that  solutions  result  in  improvements  of 
system  functions  by  selecting  and/or  developing  relevant  engineering  level 
measures  of  performance  (MOPs)  and  calculating  before/after  system 
performance. 

The  SIAP  SE  will  then  conduct  system  trade-off  analysis  of  proposed 
solutions  using  theater  force  level  models.  The  SIAP  SE  employs  theater  force 
level  models  to  determine  the  relative  improvement  in  SIAP  performance  by 
calculating  SIAP  attributes  before  and  after  changes  are  made.  Since  many 
force  level  models  also  include  capabilities  which  allow  investigation  of 
warfighting  benefits,  those  features  will  also  be  selectively  evaluated  to  help 
understand  what  warfighting  benefit  may  be  incurred  by  implementing  a 
particular  solution.  To  establish  the  value  of  a  particular  proposed  engineering 
level  change,  or  block  of  changes,  on  the  SIAP,  assessment  venues  and  related 
tools  must  be  employed  which  reveal  the  current  level  of  SIAP  performance. 

Then  the  SIAP  engineering  improvement(s)  can  be  introduced  and  the  change  in 
the  particular  measured  parameter(s)  or  metric(s)  measured.  With  sufficient 
control  over  the  experimental  variables,  the  changes  in  performance  of  the 
introduced  engineering  improvement(s)  can  be  determined.  This  type  of 
comparative  analysis  is  essential  to  the  SIAP  performance  analysis.  Trade-off 
studies  will  include  development  of  an  analysis  baseline  (“as-is”)  and  assess 
performance  changes  in  the  SIAP  as  a  result  of  the  introduction  of  alternatives  as 
developed  by  working  groups.  Tracking  the  changes  between  “as-is”  and  block 
upgrade  performance  will  enable  the  SIAP  SE  to  determine  progress  in  achieving 
the  objective  SIAP  capability. 

Using  results  from  SIAP  performance  analysis,  improvement  implementation 
cost  and  risk  estimates  provided  by  the  S/As,  and  considering  the  inter¬ 
relationships  among  the  alternatives,  the  SIAP  SE  then  investigates  how  to 
achieve  the  most  cost-effective  mix  of  proposed  solutions  to  achieve  maximum 
performance.  The  cost  and  risk  assessment  process  is  very  complex  since  it 
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could  involve  multi-systems  of  multi-Service  acquisition  efforts.  The  detail 
documentation  of  that  process  is  beyond  the  scope  of  this  technical  report.  The 
SlAP  SE  also  ensures  that  specific  solutions  contribute  to  the  objective  SIAP 
architecture  as  defined  by  architecture  modeling  and  assessment  efforts.  These 
solutions  will  then  be  incorporated  into  a  Decision  Support  Binder  (DSB)  and 
recommended  to  the  JROC. 


3.  Standard  Collaborative  Teams 

The  SIAP  SE’s  success  can  only  be  achieved  through  highly  motivated 
collaborative  teams  adhering  to  well-understood  and  disciplined  processes.  In 
response  to  this  need,  the  SIAP  SE  has  established  an  integrated  analysis  team 
to  oversee  all  SIAP  related  analysis.  Additionally,  the  SIAP  SE  will  establish  a 
Block  Working  Integrated  Product  Team  (WIPT)  for  each  block  and  working 
groups  as  necessary. 

3.1  SIAP  Analysis  Team  (SAT) 

The  SAT  is  a  SIAP  SE  led,  joint,  integrated  analysis  team  composed  of 
various  CINC/Service/Agency  (C/S/A)  analysis  experts.  The  mission  of  the  SAT 
is  to  provide  cross-Service  IADS  analysis  to  support  the  system  engineering 
decision-making.  The  SIAP  support  areas  include  HWIL,  OITL,  modeling  and 
simulation  (M&S),  and  other  venues.  Field  exercise  and  demonstration  support 
includes  planning,  data  collection,  evaluation,  root-cause  analysis,  and  after¬ 
actions  activities  required  to  system  engineer  improvements  to  the  IADS.  In  the 
M&S  area,  it  requires  support  for  development  and  evaluation  of  M&S  tools, 
operational  scenarios,  and  standard  performance  metrics  and  methodologies. 

The  SAT  does  not  replace  existing  Service  and  Agency  analysis  groups. 
Rather,  it  sets  standards  for  generating,  utilizing,  and  comparing  results  from 
multiple  SIAP  related  assessment  venues  and  related  tools  to  support  the 
collaborative  system  engineering  decision-making  process.  Products  of  the  SAT 
will  include  planning  reports  and  schedules;  documented  root-cause  analyses  of 
IADS  deficiencies,  lessons  learned,  force  capabilities  and  limitations 
assessments,  documented  analysis  results  and  engineering  recommendations. 

The  goals  of  the  SAT  are: 

1 .  Evaluate  IADS  performance  within  a  representative  architecture 

2.  Identify  performance  shortfalls  and  root  cause  of  these  shortfalls 

3.  Develop  methodologies  for  evaluation  of  candidate  solutions  for 
addressing  the  shortfalls  and  improving  IADS  performance 

4.  Evaluate  performance  of  IADS  with  proposed  solutions  implemented 

5.  Provide  recommendations  for  improving  test  venues  to  better  support 
IADS  performance  assessment  efforts. 
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The  SAT  will  support  the  disciplined  SIAP  system  engineering  process  by 
encouraging  standardization  of  all  aspects  of  analysis  required  to  support 
selected  exercises/events.  The  SAT  must  perform  collaborative,  overarching 
planning  and  scheduling  of  M&S  and  live  evaluation  resources  to  support  SIAP 
system  engineering  requirements.  Events  considered  will  be  included  in  the 
SIAP  SE  analysis  schedule  if  appropriate  support  can  be  rendered  from  SAT 
membership  and  virtual  task  force  activities. 

3.1.1  SAT  Organizational  Responsibilities 

The  SAT  shall  consist  of  three  levels  of  organizational  responsibility. 

Figure  2  depicts  these  organizational  levels.  These  three  fundamental  levels  are: 

1.  SAT  Steering  Group  (SAT  SG) 

2.  SAT  Core  Working  Group  (SAT  Core  WG) 

3.  SAT  Support  Working  Groups  (SAT  Support  WG) 


Provides  overarching  multi¬ 
event  and  programmatic 
coordination  and  decision¬ 
making 


SAT  Steering  Group: 

SIAP  Chair,  Service  Reps, 
JTAMDO,  DISA,  JNIC, 
DISA/JITC,  OSD,  JCIET,  JFCOM 


Agency  JIADS  system  level  technical 
experts  who  collaboratively  perform 
analysis  and  report  findings  on 
specific  SIAP-related  events 


Agency  technical  experts 
with  specific  expertise 
required  to  support  data 
collection,  reduction,  and 
analysis  efforts  of  SAT 
Core  WG 


SAT  Support 
Working  Groups 


Figure  2.  SAT  Organizational  Structure 
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The  following  provides  key  functions  of  the  three  SAT  organizational 

levels: 

SAT  Steering  Group 

The  following  lists  specific  elements  of  the  SAT  SG: 

.  Formal  and  long  term  designated  set  of  cognizant  senior  representatives 
from  Services  and  appropriate  Agencies 

.  SAT  SG  provides  overarching  analysis  objectives  and  logistics  support 

•  Identifies  “critical  experiments”  for  selected  exercises  and  events 

•  SIAP  SE  TF  Analysis  Branch  Head  leads  the  team,  however  Lessons 
Learned  System  Engineering  Team  (SET)  and  Block  “n”  leads  are 
members  of  the  SG 

•  Coordinates  the  activities  to  be  supported  by  the  SAT  Core  WG 
(exercises,  HWIL  tests,  simulation  studies)  with  required  C/S/A's 

•  SAT  SG  coordinates  resource  issues 

•  SAT  SG  reviews  and  endorses  final  reports  as  prepared  by  the  SAT  Core 
WG. 

The  SAT  SG  is  a  senior  level  forum  chaired  by  the  SIAP  SE  TF  Analysis 
Branch  head  and  composed  of  representatives  from  the  Architecture  and  Block 
WIPTs,  Services,  Joint  Forces  Command  (JFCOM),  Joint  Air  and  Missile 
Defense  Organization  (JTAMDO),  Joint  Interoperability  Test  Command  (JITC), 
Office  of  the  Secretary  of  Defense  (OSD)  Command,  Control,  Communications, 

&  Intelligence  (C3I),  DUSD  (Director,  Defense  Research  and  Engineering 
(DDR&E)),  Director,  Operational  Test  and  Evaluation  (DOT&E),  and  Missile 
Defense  Agency  (MDA).  The  purpose  of  the  SAT  Steering  Group  is  to  develop 
and  support  the  overall  coordination  of  exercise  and  evaluation  planning  for  the 
SIAP  SE  IAP  process.  The  SAT  SG  shall  perform  the  programmatic  functions  of 
the  SAT  effort  and  provide  the  vetting  forum  for  all  non-technical  analysis  issues. 

SAT  Core  Working  Group 

The  following  lists  key  elements  of  the  SAT  Core  WG: 

•  Formal  and  long  term  designated  set  of  joint  personnel  composed  of 
Subject  Matter  Experts  (SMEs)  from  each  major  affected  systems  are 
long  term  members  and  other  system  SMEs  as  required  (e.g.,  Center  for 
Naval  Analyses  (CNA),  JCIET,  M&S  experts,  exercise  staff) 

.  SIAP  SE  TF  member  leads  the  Core  WG  team  and  SIAP  SE  TF  provides 
infrastructural  support 

•  SAT  Core  WG  will  expand  and  contract  as  needed  to  support  the  required 
analysis  tasking. 

.  SAT  Core  WG  conducts  exercise,  HWIL  and  other  analysis  detail 
planning 
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•  SAT  Core  WG  provides  on-site  analysis  support  in  real  time  and 
immediate  post-mission  support 

•  SAT  Core  WG  convenes  to  conduct  root  cause  analysis  of  interoperability 
problems  from  exercises,  HWIL,  OITL,  etc. 

•  SAT  Core  WG  tasks  SIAP  Support  Working  Groups  for  specific  analytical 
tools/products,  data  collection,  and  bulk  statistics  generation. 

.  SAT  Core  WG  analyzes  data  from  all  sources  and  prepares  SAT 
documentation 

Ultimately,  detailed  SIAP  analysis  is  accomplished  at  the  system  level  and 
is  inherently  the  domain  of  a  very  small  set  of  SMEs  who  have  significant  system 
understanding,  joint  interoperability  understanding,  engineering  experience  and 
simulation  skills.  These  SMEs  are  largely  the  same  personnel  who  comprise  the 
Joint  Integrated  Air  Defense  System  (JIADS)  Integrated  Working  Group  (IWG). 
Membership  is  driven  by  ‘system’  level  expertise  rather  than  SERVICE/AGENCY 
expertise.  This  "Core"  team  is  the  heart  of  the  SAT  process. 

SIAP  Support  Working  Group 

The  following  lists  specific  elements  of  the  SAT  Support  WG: 

.  Ad  Hoc  basis  for  long  or  short  durations  as  required  by  tasks 

•  Specific  analysis  scope  such  as  compilation  of  bulk  statistics 

.  Membership  based  on  expertise  in  specific  tools,  etc.,  as  well  as  task  or 
effort  supported 

•  Working  groups  are  joint  endeavors  to  the  maximum  possible  extent 

There  are  also  many  SIAP  analysis  tasks  that  must  be  executed  requiring 
significant  use  of  a  particular  subject  matter  or  tool  expertise.  These  activities 
(e.g.,  creating,  executing  and  reducing  simulation  runs;  data  collection  and  data 
processing  at  exercises/events;  modifying  tools  and  producing  the  Measures  of 
Effectiveness  (MOE)  /  MOP  raw  statistics)  are  accomplished  by  a  different  set  of 
personnel  with  unique  capabilities.  These  tasks  will  be  accomplished  by  a  SAT 
Support  WG  and  will  be  overseen  by  the  system  SAT  Core  WG. 

Support  working  groups  create  data  under  the  oversight  of  the  SAT  Core 
WG,  but  they  do  not  finalize  the  analysis  or  develop  the  interpreted  report.  (E.g., 
an  Agency  such  as  Naval  Surface  Warfare  Center  (NSWC),  Corona  Division  may 
execute  the  programming  for  metric  computations,  but  the  joint  group  comes 
together  to  support  them.)  SAT  Support  WGs  provide  indispensable  service,  but 
they  do  not  have  the  responsibility  to  make  the  final  technical  decision. 

3.1.2  General  SAT  Guidelines 

The  SAT  must  lead  a  challenging  mixture  of  tasking.  The  primary  task  of 
the  SAT  is  to  perform  collaborative  SIAP  related  analysis  through  M&S  and 
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open-air  exercises  and  events  to  support  recommendations  for  improvements  to 
IADS  warfighting  shortfalls.  A  secondary  but  mutually  supportive  task  is  to 
coordinate  and  budget  for  those  same  joint  and  Service-specific  evaluations  such 
that  results  from  one  venue  and  related  tool  can  be  used  to  support  results  from 
another.  This  primary  task  may  only  be  achieved  through  an  apolitical  agenda- 
free  environment.  The  secondary  task  is  froth  with  organizationally  partisan 
issues.  Coordination  of  these  issues  creates  a  potentially  confrontational 
environment  that  could  run  counter  to  collaborative  engineering. 

While  the  two  tasks  must  be  mutually  supportive,  vetting  the  political 
agendas  must  occur  outside  the  task  of  performing  joint  analysis.  Therefore,  the 
programmatic  issues  must  be  coordinated  at  a  senior  level  without  squashing  the 
free  exchanges  between  the  system  SMEs  at  the  Core  WG  level.  High  level  of 
corporate  trust  is  essential  to  accomplish  the  collaborative  analysis  tasking.  This 
analysis  work  cannot  be  influenced  by  the  program  management  level 
interaction,  discussion  of  resources  (or  lack  of  resources),  academic  musings  on 
the  “big  picture,”  unique  agendas,  etc.  These  programmatic  interactions  shall 
occur  at  the  SG  level  only.  As  is  the  case  in  the  JIADS  IWG  data  analyses,  the 
SAT  must  focus  on  the  data  at  hand  and  the  way  systems  currently  work  and 
without  too  much  distraction  on  the  future  architectural  concepts.  Therefore,  the 
SAT  SG  shall  be  established  to  control  the  overall  process  and  provide  the 
required  buffer  between  real  work  and  politics. 

SAT  Core  WG  members  shall  accomplish  the  analysis  and  write  the  report 
to  describe  what  happened.  Socialization  and  vetting  of  political  issues  will  occur 
at  the  SAT  SG  level  AFTER  the  results  are  developed.  All  stakeholders  must 
jointly  develop  SAT  results  with  open  and  robust  participation.  Interim  findings 
are  not  elevated  to  outside  agencies  or  high-level  management  until  there  is  a 
technical  consensus  among  the  Core  WG  SMEs.  The  Core  WG  is  the  real-world 
filter  for  all  reduced  data. 

SAT  Core  WG  SMEs  may  be  contractors  or  government  personnel.  SMEs 
may  work  directly  for  program  offices  or  for  Service  labs.  All  personnel  must  be 
freely  empowered  to  work  across  agencies  on  a  technical  level. 

IADS  SMEs  have  developed  their  credentials  (and  maintain  them)  by 
performing  full-time  jobs  for  their  C/S/A  sponsors.  The  SIAP  SE  TF  cannot  build 
new  SMEs  at  this  level  in  anything  less  than  years  (and  even  then,  by  the  nature 
of  the  beast,  new  SMEs  would  evolve  out  of  the  systems  rather  than  joint 
Agencies).  The  SAT  must  be  sensitive  to  the  potential  of  alienating  SMEs’ 
sponsors  who  may  ultimately  control  priorities  for  their  time.  The  SAT  SG  must 
create  an  environment  that  the  SMEs  individually  desire  to  participate  in  and 
must  minimize  time  and  travel  demands  to  avoid  conflicts. 

Good  infrastructure  is  important  to  maintaining  the  appropriate 
professional  environment  and  is  a  serious  job  for  the  SAT  SG.  A  smoothly 
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running,  well  laid  out  “Fishbowl”  with  plenty  of  electrical  power,  good  lighting, 
security’ support,  etc.  makes  a  tremendous  difference,  especially  when  the  joint 
"team"  is  working  nearly  around  the  clock  during  on-site  mission  support.  This 
will  win  hearts  and  minds  of  the  SME's  and  provide  excellent  press  for  the 
inevitable  crowd  of  VIPs.  It  will  be  the  responsibility  of  the  SAT  SG  to  provide 
resources  to  support  the  professional  infrastructure  required  to  perform  the  SAT 
functions. 

3.2  Block  Working  Integrated  Product  Team  (WIPT) 

The  Block  WIPT  is  comprised  of  the  Block  Manager,  SIAP  SE  engineers 
and  Service  SMEs  who  will  be  charged  with  developing  the  block  system 
engineering  recommendations.  They  will  apply  an  established  IEEE  STD  1220- 
1998  disciplined  system  engineering  process  (requirements  analysis,  functional 
analysis,  synthesis  of  alternatives,  system  analysis,  and  cost  benefit  analysis)  to 
determine  the  most  beneficial  way  to  provide  improvements  in  warfighting 
capabilities. 

The  Block  WIPT  will  be  a  formal  body  of  long  term,  designated  joint 
personnel  and  SMEs  from  each  major  affected  system.  The  Block  WIPT  may 
rely  on  other  Service/Agency  SMEs  as  required  in  addition  to  support  from  the 
SAT  working  groups.  The  Block  Manager  leads  the  team  and  SIAP  SE  TF 
provides  infrastructure  support. 

Block  WIPT  responsibilities  that  ensure  a  disciplined  approach  is  taken 
include: 

.  Develop  the  problem  statements  and  tasks  for  the  block  issues 
.  Develop  the  Statement  Of  Work  (SOW)  for  the  block  support  tasks 
.  Develop  the  block  Plan  Of  Actions  and  Milestones  (POA&M) 

.  Oversee  the  block  system  engineering  effort  conducted  in  accordance 
with  the  SOW 

.  Manage  the  block  master  schedule  and  ensure  integration  of  all  detailed 
schedules  for  the  block  effort 

•  Publish  and  maintain  the  block  status  brief  to  be  briefed  to  the  SIAP 
Integrating  Integrated  Product  Team  (IIPT) 

.  Establish  Block  Working  Groups  to  engineer/investigate  issues 
.  Work  with  the  SIAP  core  engineering  team  and  analysis  branches  to 
publish  the  DSB 

.  Draft  the  Block  Improvement  Plan  based  on  the  DSB  built  by  the  block 
engineering  effort 

.  Drafting  minutes  of  meetings  as  well  as  document  follow-up  action 
activities 

.  Update  and  maintain  block  information  and  data  on  the  SIAP  worksite 
.  Oversee  theater  level  analysis  at  attribute  and  MOE  level 
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3.3  Block  Working  Groups 

The  SIAP  block  system  engineering  approach  embraces  a  number  of 
specific  analysis  problem  areas  for  each  new  block  of  issues.  As  necessary, 
working  groups  will  be  established  when  assigned  tasks  require  significant 
subject  matter  or  tool  expertise.  Block  Working  Groups  will  be  composed  of 
SIAP  SE  TF  staff  members  and  SMEs  from  government  and  industry  that  are 
part  of  the  larger  SIAP  SE  ‘virtual’  TF.  A  number  of  SAT  analysis  related 
activities  (e.g.,  creating,  executing  and  reducing  simulation  runs;  data  collection 
and  processing  at  exercises;  modifying  tools  and  producing  MOE/MOP  raw 
statistics)  are  the  purview  of  the  Block  Working  Groups. 

Overarching  responsibilities  of  Working  Group  members  will  include: 

.  Develop  requirements  and  functional  analysis  of  assigned  issue 
areas 

.  Work  with  theater  level  modelers  to  develop  interface  (i.e.,  ‘hooks’) 
to  engineering  level  model  inputs 

.  Develop  solution  options  for  performance  assessment 


4.  Standard  Tools 

The  SIAP  SE  employs  multiple  test  events,  evaluation  scenarios  (Common 
Reference  Scenarios  (CRS)),  and  standard  metrics  to  produce  analyses  that  are 
comparable,  operationally  representative,  repeatable,  and  meaningful. 

4.1  Venues 

Many  modeling  &  simulation,  HWIL,  and  OITL  analysis  venues  exist  today. 
These  existing  resources  are  used  by  the  Services  and  Joint  organizations  to 
provide  an  analytical  basis  for  design,  development  and  evaluation  of  Theater  Air 
and  Missile  Defense  (TAMD)  systems.  System  specific  and  joint  integrated  tools 
provide  a  broad  range  of  analysis  capabilities  at  various  measurement  levels. 
These  separate  modeling  and  simulation  tools  (i.e.  JDN  networks  tools  and  IADS 
modeling  tools)  currently  exist  and  model  different  areas  of  interest  for  SIAP 
analysis.  No  one  tool  can  measure  the  interoperability  of  the  IADS. 
Interoperability  is  reflected  in  warfighting  performance  metrics,  which  provide  an 
indication  of  how  well  the  Family  of  Systems  (FoS)  is  supporting  the  warfighter. 
By  federating  several  models  and  analytic  constructs  to  support  parametric 
measurements  at  the  system  level  (e.g.  Air  Defense  Simulation  (ADSIM)  with 
Extended  Air  Defense  Simulation  (EADSIM)),  variations  in  system  functional 
performance  can  be  traced  to  force  level  capability  improvements. 

Of  the  many  potential  assessment  venues  available,  there  has  not  been  an 
attempt  to  coordinate  or  synergize  across  the  multi-Service  venues  in  the  past. 
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The  result  is  that,  while  many  were  characterizing  SIAP  related  performance  in 
similar  ways,  none  are  common  enough  to  equitably  compare  results  from  one 
venue  with  another,  or  to  build  a  consistent  cumulative  story  of  the  SIAP 
performance  based  on  their  multiple  contributions. 

To  equitably  relate  results  of  various  venues  and  related  tools  with  one 
another,  it  is  necessary  to  standardize  both  a  set  of  SIAP  metrics  and  evaluation 
scenarios  to  provide  as  much  consistency  in  the  resulting  measurements  as 
possible.  The  SIAP  SE  took  this  approach  and  worked  with  the  appropriate 
stake  holding  S/As  to  develop  a  standardized  set  of  metrics,  which  could  be  used 
across  all  of  the  proposed  assessment  venues  and  related  tools.  In  addition,  to 
ensure  that  assessments  (where  feasible)  are  carried  out  in  the  same  or  similar 
operational  contexts,  several  CRS  are  proposed  and  are  being  jointly  developed 
by  the  S/As  under  SIAP  SE  coordination. 

The  level  of  standardization  introduced  across  the  many  potential 
assessment  venues  facilitates  utilization  of  all  available  data  collected  by  many 
different  DoD  activities,  and  allows  the  SIAP  SE  to  leverage  the  considerable 
efforts  of  many  joint  DoD  activities  for  minimal  extra  effort  and  expense. 

The  following  paragraphs  briefly  describe  some  of  the  primary  venues, 
which  will  be  utilized  for  SIAP  analysis,  citing  some  of  the  major  strengths  and 
weaknesses  of  each. 

4.1.1  Live  Exercises 

Live  exercises  such  as  JCIET  and  Roving  Sands  employ  the  use  of  actual 
system  hardware  and  software  and  thus  provide  the  best  representations  of 
legacy  system  functionality.  We  would  also  expect  that  live  exercises  provide  the 
best  characterization  of  warfighting  performance  of  the  FoS.  However,  limited 
availability  of  assets  may  preclude  supporting  an  exercise  with  the  exact  platform 
configurations  desired.  Conduct  of  before/after  comparative  analysis  requires 
having  systems  that  can  operate  in  either  legacy  or  upgrade  mode,  or  the 
availability  of  separate  systems,  some  of  which  are  upgraded  and  some  of  which 
are  not,  in  order  to  compare  their  performance  with  each  other.  Many  platforms 
have  multiple  baselines,  and  may  be  in  various  stages  of  upgrade,  so  that  the 
configurations  available  may  not  be  entirely  representative  of  the  group  as  a 
whole. 

In  addition,  live  exercises  generally  are  inherently  expensive,  require  long 
lead  times  to  set  up,  are  subject  to  many  uncontrolled  variables,  and  are  not 
amenable  to  multiple  repetitions  of  a  particular  test  to  assess  the  effects  of  a 
particular  change.  Additional  limitations  of  live  exercises  include  networking 
restrictions  on  Link  16  operation,  unrealistic  network  threat  loading,  and  often 
restricted  airspaces  that  can  impact  the  participation  of  friendly  and  hostile  forces 
when  compared  to  actual  wartime  employment.  Finally  the  warfighting  realism  of 
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live  exercises  is  affected  by  public  safety  and  environment  restrictions  on  live 
operations. 

Live  exercises  can  be  extremely  useful  to  baseline  real  system 
performance  and  to  provide  real-world  data  in  order  to  validate  results  from  other 
venues,  especially  M&S  results.  Future  events  support  analysis  of  system 
performance  after  changes  to  the  systems  are  implemented.  Each  participating 
system  can  record  data  for  post-event  root  cause  analysis.  The  amount  and  type 
of  data  can  vary  from  system  to  system  as  well  as  event  to  event. 

While  many  factors  influence  the  amount  and  quality  of  the  data  available 
(e.g.  equipment  failures,  weather,  number  and  type  of  platforms  participating  in 
the  events),  empirical  analysis  can  provide  representative  information  from  real 
system  hardware,  operated  by  real  warfighters,  in  realistic  engagements  and 
therefore  represents  very  credible  exhibitions  of  SIAP  performance.  So-called 
data-driven  modeling  tools  such  as  those  used  by  CNA,  e.g.,  Operational  Data 
Driven  for  Correlation  Algorithm  Performance  Evaluation  (ODDSCAPE)  and 
Naval  Surface  Warfare  Center,  Corona  Division’s  Performance  Evaluation  Tool 
(PET),  support  the  evaluation  of  live  exercises. 

4.1.2  Hardware-in-the-Loop 

Tools  in  the  HWIL  venue,  such  as  the  JDEP,  attempt  to  retain  the  fidelity 
of  real  hardware  and  software-in-the-loop  by  linking  real  or  laboratory  systems  in 
a  controlled  environment,  typically  without  radio  frequency  (RF)  transmissions. 
This  permits  better  control  and  repetition  of  experiments,  use  of  specific  desired 
hardware  and  software  configurations,  and  the  ability  to  do  more  reliable 
before/after  comparisons.  FoS  analysis  is  somewhat  limited  with  most  of  the 
tools  in  this  venue  due  to  the  limited  numbers  of  platforms  participating  in  an 
event.  In  addition,  HWIL  tools  are  often  extremely  limited  in  their  capability  to 
replicate  sensor,  communications,  and  environmental  issues. 

4.1.3  Operator-in-the-Loop 

The  OITL  venue  tools,  such  as  the  VWC  and  at  the  Joint  National 
Integration  Command’s  (JNIC)  test  facility,  are  cost-effective  ways  to  evaluate 
integrated  system  environments  with  respect  to  the  man-machine  information 
interface.  These  tools  provide  a  very  important  ingredient  for  FoS  evaluation 
e.g.,  operator,  SIAP  related  interaction  and  decision-making  effects.  This  effect 
is  a  significant  element  of  evaluating  the  military  utility  of  information  sharing 
improvements  at  the  system  level.  JTAMDO  has  successfully  utilized  the  VWC 
in  assessing  warfighting  effectiveness  given  selected  SIAP  attribute  performance 
levels.  JNIC’s  Wargame  2000  has  provided  similar  results  for  TAMD  missions. 

However,  OITL  tools,  like  HWIL,  generally  are  limited  in  their  capability  to 
fully  replicate  family  of  systems  functionality.  While  HWIL  tools  permit  better 
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control  and  repetition  of  experiments,  by  using  specific  desired  hardware  and 
software  configurations,  and  the  ability  to  do  more  reliable  before/after 
comparisons,  OITL  repeatability  is  somewhat  difficult  to  achieve. 

4.1.4  Digital  Modeling  and  Simulation  (M&S) 

Digital  M&S  tools  serve  a  wide  spectrum  of  purposes  that  can  range  from 
design  to  operational  effectiveness  assessments.  Consequently  models  and 
simulations  exist  with  differing  levels  of  detail  suited  to  their  particular  application. 
Different  levels  of  functional  assessments  form  what  may  be  called  a  hierarchy  of 
models  and  simulations  as  shown  below  (figure  3). 

RESOLUTION  FUNCTIONS  SUPPORTED  MODELS  &  SIMULATIONS  FORCE  OR  SYSTEM  LEVEL 

Increasing 


Figure  3.  Hierarchy  of  Models  and  Simulations 


4.1 .4.1  Engineering  Modeling  and  Simulation 

At  the  engineering  level,  models  indicate  individual  system  and  FoS 
performance  capabilities,  or  system  MOPs  and  SIAP  attributes.  For  the  SIAP  SE 
in  Block  0,  this  meant  addressing  current  system/subsystem  performance  issues 
such  as  formation  tracking  and  identification  (ID)  taxonomy.  For  Block  1 ,  this 
means  the  reduction  of  dual  tracks,  improvement  of  combat  ID,  Theater  Ballistic 
Missile  Defense  (TBMD)  performance,  and  data  sharing. 

While  engineering  level  modeling  may  not  be  required  for  all  analyses 
(e.g.,  determining  requirements  such  as  force  structure  assessments/alignments 
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or  logistics  flows); it  is  essential  for  the  SIAP  SE  because  it  is  precisely  these 
engineering  level  issues  that  the  SIAP  SE  has  been  directed  to  address.  Time 
and  time  again,  it  has  been  shown  that  engineering  level  issues  issues  have 
impacted  the  warfighter  at  the  operational  level.  In  addition,  since  the  SIAP  SE 
will  conduct  modeling  at  higher  levels,  it  must  1)  be  able  to  determine  the  amount 
of  improvement  (MOP)  in  each  block,  and  2)  be  able  to  aggregate  these 
improvements  such  that  they  can  be  characterized  in  model  runs  at  the 
mission/battle  and  theater/campaign  levels. 

4.1. 4.2  Engagement  Modeling  and  Simulation 

At  the  engagement  level,  models  are  used  to  evaluate  system 
effectiveness  against  threat  systems.  Typically  expressed  as  MOEs,  the 
engagement  level  is  generally  used  in  “one  versus  one”  or  “few  versus  few”  type 
of  engagements.  While  this  definition  works  well  for  a  weapon  system,  the  issue 
is  much  more  obscure  when  addressing  effectiveness  within  a  SIAP  context 
simply  because  SIAP  issues  cannot  be  addressed  based  upon  the  effectiveness 
against  the  destruction  or  suppression  of  a  particular  threat  system,  i.e.  SIAP  is 
not  an  element  of  the  kill  chain.  Rather,  it  is  the  enabler  for  the  kill  chain 
sequence.  Consequently,  while  the  SIAP  attributes  and  MOEs  of  block 
improvements  cannot  be  based  intrinsically  upon  known  threat  capabilities  and 
raid  densities,  it’s  attributes  can  be  measured  based  upon  the  block 
improvements  and  by  corollary,  its  ability  to  support  tracking  and  engagement 
functions  where  and  when  required;  i.e.,  clarity,  continuity,  completeness.  The 
resulting  MOE  at  the  engagement  level  should  reflect  less  fratricide,  increased 
detection  ranges,  optimized  engagement  times,  and  optimized  weapons 
employment  opportunities.  All  of  which  ultimately  allow  friendly  forces  to  disrupt 
enemy  decision  cycles  and  preempt  hostile  action. 

4.1 .4.3  Mission/Battle  Modeling  and  Simulation 

At  the  mission/battle  level,  models  allow  results  gained  at  the 
engagement  level  to  be  aggregated  to  a  force  level.  This  aggregation  would 
encompass  a  multi  platform  force  package  designed  to  accomplish  a  specific 
mission  objective  such  as  air  superiority,  interdiction,  or  Suppression  of  Enemy 
Air  Defenses  (SEAD).  While  all  levels  of  modeling  are  important,  it  is  at  this  level 
where  the  benefits  of  the  block  improvements  really  begin  to  manifest 
themselves  because  the  entire  objective  is  to  demonstrate  warfighter  benefit  at 
the  force  level  based  upon  Cost  and  Operational  Effectivess  Analysis  (COEA), 
compatibility,  and  interoperability. 

4.1 .4.4  Theater/Campaign  Modeling  and  Simulation 

At  the  theater/campaign  level,  models  represent  combined  force 
operations  and  are  used  in  theater  or  campaign  level  conflicts  to  determine  the 
long  term  outcome.  Theater/campaign  level  models  are  also  used  to  determine 
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force  structure  assessments/alignments  or  logistics  flows.  For  the  SIAP  SE, 
models  at  this  level  will  be  used  to  determine  how  the  block  improvements  affect 
the  air  picture  at  the  Joint  Task  Force  (JTF)  level.  Results  from  engineering  level 
model  runs  will  be  aggregated.  These  aggregate  performance  parameters  will 
then  be  characterized  in  the  theater/campaign  level  model. 

The  best  way  to  achieve  robust  parametric  evaluations  of  system  of 
system  performance  with  the  potential  to  dynamically  evaluate  force  on  force 
engagement  effects  is  through  digital  M&S.  The  primary  strength  of  available 
digital  simulations  is  the  capability  to  permit  expansion  limited  engagement 
vignettes  to  force-on-force  and  theater  level  analysis.  This  scale  up  can  be  done 
in  a  controlled,  repeatable  environment,  with  the  ability  to  execute  multiple  runs 
with  many  individual  parameter  variations  representing  many  different  potential 
systems  engineering  fixes  or  improvements.  However,  while  digital  M&S 
provides  the  capability  to  represent  large  numbers  of  systems  in  repeatable 
scripted  runs,  it  is  limited  in  the  precision  with  which  individual  system 
performance  can  be  replicated.  The  greatest  loss  in  moving  from  real  hardware 
and  software  components  in  HWIL,  OITL,  and  Live  venues,  is  in  the  realistic 
representation  of  system  functionality. 

Digital  simulations  can  be  used  at  different  analysis  levels  to  characterize 
and  emulate  system  and  subsystem  performance  in  terms  of  MOPs,  to  evaluate 
the  effect  those  changes  in  system  MOPs  have  on  higher  level  SIAP  attributes, 
and  to  evaluate  the  effect  that  the  changes  of  SIAP  attributes  have  on  warfighting 
MOEs.  A  combination  of  specific  system  functional  analysis  and  parametric 
analysis  must  be  done.  For  example,  theater  level  parametric  assessments  can 
be  made  with  theater  level  modeling  tools  to  evaluate  the  effects  of  different 
levels  of  system  performance  on  SIAP  attributes  (e.g.,  assume  different  levels  of 
navigation  accuracy  for  each  participating  unit  and  assess  the  resulting  SIAP). 

This  type  of  tool  is  useful  in  defining  ranges  of  acceptable  subsystem 
performance  that  needs  to  be  obtained  to  achieve  a  certain  level  of  SIAP.  Other 
more  detailed  M&S  might  then  use  high  fidelity  system  functional  representations 
(both  legacy  and  proposed  upgrades)  to  determine  the  actual  values  at  which  the 
system  or  subsystem  is,  or  might  be,  capable  of  operating.  Insertion  of  actual 
predicted  performance  values  into  higher-level  models  then  allows  investigation 
of  the  behavior  of  the  SIAP  with  specific  sets  of  potential  change  options 
implemented. 

4.1 .4.5  Primary/Secondary  SIAP  Functions 

In  any  modeling  environment,  the  required  fidelity  of  a  model’s  specific 
system  representations  depends  on  the  specific  purpose  of  the  analysis.  For 
SIAP  analysis  purposes,  system  functions  fall  into  two  broad  categories. 

.  Primary  system  functions  are  those  that  need  to  be  modeled  at  relatively 

high  fidelity  because  they  are  to  be  varied  in  the  course  of  the  analysis  to 
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show  the  effects  of  different  implementations.  For  example,  conducting  a 
before/after  assessment  of  different  implementations;  e.g.,  modeling  of 
current  legacy  sensor  alignment  process  or  its  result  when  a  new  sensor 
alignment  process  is  to  be  evaluated  and  compared  with  the  current 
function.  Primary  functions  need  to  be  represented  at  a  sufficient  level  of 
fidelity  that  changes  to  those  functions  reliably  represent  changed 
behavior  of  the  system  so  that  credible  recommendations  for  system 
changes  can  be  made. 

.  Secondary  functions  are  those  whose  inputs  are  required  for  conduct  of  a 
particular  analysis,  but  which  need  not  be  varied,  so  that  only  a  constant 
representative  implementation  is  needed  (e.g.,  a  generic  tracker  to 
generate  representative  inputs  to  a  track  file,  when  a  new  correlation 
algorithm  is  being  evaluated,  provided  the  generic  tracker  generates  all 
data  required  by  the  candidate  correlation  algorithm(s)).  The  fidelity  of 
secondary  system  representations  will  not  be  considered  sufficient  to 
draw  conclusions  with  respect  to  actual  system  performance,  nor  to  make 
engineering  change  recommendations  to  those  systems  or  system 
functions. 

When  non-network  issues  are  being  investigated  (e.g.,  sensor  alignment, 
geodetic  registration,  system  correlation),  high  network  fidelity  is  not  required.  In 
such  a  case,  the  network  is  a  secondary  function  and  a  model  with  a  generic  low 
or  medium  fidelity  data  link  representation  may  be  adequate.  However,  when 
investigating  particular  SIAP  degrading  effects  associated  with  network  overload, 
for  example,  the  network  and  the  data  link  equipment  that  create  the  network  are 
primary  systems,  and  high  network  fidelity  is  required  to  provide  an  accurate 
representation  of  the  network  effects  on  SIAP. 

The  ultimate  goal  of  the  SIAP  SE  digital  M&S  development  is  to  create  a 
modeling  environment  in  which  specific  engineering  changes  can  be  inserted  into 
specific  platform  representations,  in  a  theater  level  operational  environment,  and 
to  characterize  the  changes  in  SIAP  and  IADS  performance  that  result. 

4.2  Common  Reference  Scenarios  (CRS) 

Perhaps  the  most  important  task  in  establishing  a  disciplined  analytical 
process  and  evaluating  IADS  performance  is  the  development  of  scenarios  for 
that  analysis.  To  level  the  playing  field  among  different  analysis  venues,  a 
common  operational  laydown,  or  frame  of  reference,  must  be  used  as  an  input. 
This  baseline  frame  of  reference  would  minimize  the  number  of  variables  among 
the  venues  by  ensuring  the  IADS  is  measured  within  the  same  environment  (e.g., 
same  radar  angles  to  threat,  communications  geometry,  force-flows,  etc).  If 
necessary,  minor  excursions  can  be  made  upon  this  baseline  in  order  to  examine 
friendly  and  threat  interactions  specific  to  a  block  issue  area. 
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Scenario-based  design  analysis  is  recognized  as  a  significant  tool  to 
support  the  system  engineering  process.  “Scenarios  afford  multiple  views  of  an 
interaction,  diverse  kinds  of  and  amounts  of  detailing,  helping  developers 
manage  the  many  consequences  entailed  by  any  given  design”  [Carroll  ].  The 
SlAP  SE  is  extending  the  concept  of  scenario-based  design  by  supporting  the 
contextual  representation  of  the  SlAP  operational  concept.  A  representative 
operational  context  supports  evaluation  of  IADS  capabilities  in  real-world 
replicated  settings.  The  operational  context  provides  a  foundation  for  the 
development  of  the  Operational  View  of  the  TAMD  Integrated  Architecture. 

The  SlAP  SE  disciplined  process  is  based  on  standardization  of  key 
simulation  tools  and  scenarios,  providing  common  reference  frames  for  “apples- 
to-apples”  comparisons.  In  addition  common  reference  frames  support  joint 
evaluation  of  the  IADS  in  both  the  simulated  and  live  environments.  The 
baseline  frame  of  reference  must  bear  some  resemblance  to  the  possible  IADS 
operational  environments.  One  of  these  standardized  tools  is  the  establishment 
of  a  set  of  CRS  to  encompass  a  broad  spectrum  of  operational  environments. 

Three  CRS  representing  potential  regional  conflicts  have  been  defined 
based  on  the  DoD  Defense  Planning  Guidance  process.  The  first  CRS  was 
created  for  the  2003-05  timeframe  and  jointly  endorsed  through  the  JTAMD 
process.  The  CRS  is  composed  of  digitally  scripted  hostile  and  friendly  force 
dynamic  interactions  in  operationally  significant  time-phased  campaigns. 
Engineering  vignettes,  extracted  from  the  CRS,  provide  particular  platform 
engagements  of  interest,  which  can  be  applied  to  the  M&S,  HWIL,  OITL,  and  live 
exercises  to  support  repeatable  engineering  level  FoS  assessments.  Within  this 
framework,  evaluations  of  SlAP  system  enhancements  and  the  resulting  impact 
on  warfighter  capabilities  from  the  system/unit  through  the  force-on-force  level 
may  be  quantified.  (See  SlAP  Technical  Report  2002-  Common  Reference 
Scenarios  (CRS)  and  related  appendices  for  more  information.) 

4.3  Networks 

The  SlAP  SE  has  coordinated  with  the  Service  Network  Design  Facilities 
(NDFs)  to  provide  up  to  eight  Link  16  network  designs  for  SlAP  related  analyses. 
The  network  designs  are  required  to  define  the  technical  network  interface 
requirements  of  the  supporting  JDN  systems.  Initially  the  network  designs  will  be 
Link  16  (and  Link  1 1  where  essential)  only,  not  the  whole  theater  multi-Tactical 
Digital  Information  Link  (TADIL)  networks.  They  will  be  used  primarily  to  support 
the  high  fidelity  Link  16/1 1  modeling  performed  by  such  models  as  the  ADSIM. 
However,  they  will  be  available  to  all  modeling  points  of  contacts  (POCs),  who 
can  extract  the  information  necessary  for  their  model.  If  SlAP  analysis  of  a 
specific  block  issue  requires  modifications  to  a  network  design,  the  assigned 


1  John  M.  Carroll  is  Virginia  Tech  Professor  and  Director  of  its  Center  for  Human-Computer 
Interaction.  He  is  a  leading  expert  on  scenario-based  design,  has  spoken  at  various  related 
seminars  and  written  books  on  the  subject. 
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modeling  center  may  first  coordinate  with  the  SIAP  SE  to  create  variants,  and 
then  submit  these  derivative  network  designs  to  the  Service  NDFs  for  review  and 
approval.  The  Service  NDFs  will  then  review  for  correctness  and  validate  the 
modeling  center  developed  network  design. 

4.4  Metrics 

A  critical  part  of  system  engineering  the  SIAP  is  identifying  a  standardized 
quantification  of  performance.  Metrics  are  used  to  objectively  evaluate  the  ability 
of  candidate  approaches  to  meet  JROC-validated  Capstone  requirements. 
Additionally,  they  allow  us  to  understand  how  we  are  progressing  toward  the 
objective  end-state. 

The  concept  of  a  SIAP  lends  itself  to  quantifiable  warfighting  MOEs, 
mission  level  attributes,  and  system  level  MOPs.  Some  of  these  values  have 
been  defined,  so  realistic  assessment  strategies  to  evaluate  compliance  can  be 
developed.  SIAP  Key  Performance  Parameters  (KPPs)  have  been  defined  in  the 
TAMD  and  Combat  Identification  (CID)  Capstone  Requirements  Documents 
(CRDs).  These  operational  requirements  must  be  translated  in  a  traceable  way 
into  lower-level  technical  requirements  that  can  be  used  by  the  disciplined 
system  engineering  process,  and  to  objectively  assess  progress  in  achieving  the 
required  SIAP  capability.  However,  as  indicated  in  the  SIAP  SE  TF  Charter,  one 
of  the  SIAP  SE’s  jobs  is  to  help  evolve  the  definition  of  SIAP. 

Quantifiable  and  testable  MOEs,  attributes,  and  MOPs,  are  the  linchpin  to 
the  SIAP  system  engineering  efforts.  MOEs  and  MOPs  must  support  various 
analysis  methods  including  sensitivity  analyses  to  support  technical  trade-offs, 
modeling  and  simulation,  experimentation,  land-based  test  and  evaluation  (such 
as  JDEP),  interoperability  certification  (such  as  that  provided  by  JITC),  and 
evaluation  in  an  operational  context  (such  as  JCIET)  of  SIAP  related  changes 
and  other  warfighting  capability  improvements.  Several  efforts  have  been 
undertaken  to  develop  a  quantifiable  set  of  SIAP  related  measures. 

Such  measures  provide  answers  to  three  fundamental  questions: 

.  What  do  we  have  today?  (Evaluative  measures) 

.  What  is  required?  (Predictive  measures) 

•  How  do  we  get  what  we  need?  (Prescriptive  measures) 

Quantifying  answers  to  these  questions  provides  an  analysis  roadmap  for 
system  improvement.  Ultimately  these  types  of  measurements  must  be 
evaluated  at  various  levels  of  aggregation  i.e.,  MOPs  at  the  system/platform 
level,  attributes  at  mission/effectiveness,  theater,  and  force  level  and  MOEs  at 
the  force-on-force/Campaign  level.  These  levels  determine  a  hierarchy  of 
quantifiable  characteristics  as  shown  in  Figure  4.  The  flow-down  of  quantifiable 
measures  from  MOEs  at  the  force  level  to  system  level  MOPs  provide  the 

Page  24 
9  July,  2002 


capability  to  determine  how  systemic  problems  and  improvements  affect 
warfighting  capability. 

To  build  a  common  lexicon,  and  make  progress  toward  achieving  the  SIAP, 
it  is  critical  that  the  processes  and  products  that  result  from  the  various  measures 
and  attributes  efforts  converge  to  a  standardized  approved  set.  At  a  minimum,  a 
standard  set  of  definitions  and  derivations  of  SIAP  attributes  must  be  used 
across  Services  and  joint  organizations.  These  attributes  provide  a  common 
reference  to  measure  a  SIAP.  In  addition,  the  appropriate  MOEs  and  MOPs  must 
be  identified  and  used  by  testers,  analyzers  and  evaluators  such  that  common 
criteria  may  be  used  to  evaluate,  predict,  and  prescribe  performance. 


Figure  4.  MOP/MOE  Mapping 

4.4.1  Engineering  Level  Metrics:  Measures  Of  Performance  (MOPs) 

At  the  engineering  level,  each  potential  change  will  have  a  number  of 
specific  engineering  level  MOPs  that  can  be  defined  to  characterize  the  relative 
performance  of  a  specific  improvement.  Consequently,  no  attempt  has  been 
made  to  develop  a  master  list  of  MOPs  across  all  SIAP  assessment  venues. 
Working  groups  are  responsible  for  developing  proposed  system  upgrades 
associated  with  their  particular  block  issues.  In  conjunction  with  their  proposed 
upgrades,  they  will  need  to  define  the  measures  by  which  the  anticipated 
improved  performance  will  be  judged  at  the  engineering  level.  These  working 
groups  will  also  have  to  help  define  the  relationships  between  their  proposed 
engineering  MOPs  and  various  SIAP  attributes  so  the  effects  of  the  engineering 
level  changes  can  be  understood  in  terms  of  SIAP  impact  at  the  attribute  level. 
Table  1  lists  representative  MOPs.  (See  SIAP  Technical  Report  2001-002 
Measures  of  Effectiveness  (MOEs).  and  Measures  of  Performance  (MOPs)  for 
more  information.) 
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Table  1 .  Representative  IADS  MOPs 


Time  difference  between  system  internal  time  at  central  track  stores  and  JTIDS 
terminal 

Latency  of  messages  due  to  buffering,  prioritization,  staieness,  and  time  slot 
allocation 

Translational  and  rotational  error  quantities 

Percent  of  time  units  correctly  report  track  quality  in  conformance  with 
MIL-STD-6016 


4.4.2  SIAP  Attributes 

Under  the  leadership  of  the  SIAP  SE,  Service/Agency  SMEs  have 
developed  a  rigorously  defined  set  of  SIAP  performance  attributes.  The  effort 
included  specific  procedures  for  implementing  and  using  the  attributes.  SIAP 
attributes  define  and  characterize  IADS  performance  in  terms  that  are  directly 
related  to  the  TAMD  and  CID  CRD  KPPs.  While  the  CRD  KPPs  are  oriented 
toward  theater-wide  performance  values,  the  SIAP  attributes  also  include 
methods  for  characterizing  IADS  performance  at  the  individual  unit  level.  Table  2 
lists  the  defined  set  of  SIAP  attributes  for  air  breather  aerospace  objects.  The 
SIAP  SE  is  currently  working  to  definite  SIAP  attributes  for  ballistic  missile 
aerospace  objects  as  part  of  the  Block  1  effort.  (See  SIAP  Technical  Report 
2001-001  Attributes  and  SIAP  Technical  Report  2002-xxx  Ballistic  Missile  Single 
Integrated  Air  Picture  (SIAP)  Metrics  for  more  information.) 

Table  2.  SIAP  Attributes 


Completeness:  The  air  picture  is  complete  when  all  objects  are  detected,  tracked,  and 
reported  _ _ _ _ 

Clarity:  The  air  picture  is  clear  when  it  does  not  include  ambiguous  or  spurious  tracks. _ 

Continuity:  The  air  picture  is  continuous  when  the  track  number  assigned  to  an  object  does  not 
change. _ _ 

Kinematic  Accuracy:  The  air  picture  is  kinematically  accurate  when  the  position  and  velocity 
of  a  track  agrees  with  the  position  and  velocity  of  the  associated  target.  _ 

ID  Completeness:  The  ID  is  complete  when  all  tracked  objects  are  labeled  in  a  state  other 
than  “unknown’’. _ 

ID  Accuracy:  The  ID  is  accurate  when  all  tracks  are  labeled  correctly.  _ 

ID  Clarity:  The  ID  is  ambiguous  when  a  tracked  object  has  two  or  more  conflicting  ID  states. 

Commonality:  The  air  picture  is  common  when  the  tracks  held  by  each  participant  have  the 
same  track  number,  position,  and  ID. _ _ _ 
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4.4.3  Military  Utility  Metrics:  Measures  of  Effectiveness  (MOEs) 

The  SIAP  SE  TF  is,  by  design  and  charter,  an  engineering  organization, 
but  it  is  also  chartered  with  making  recommendations  to  the  JROC  with  respect 
to  what  engineering  changes  should  be  implemented  across  a  broad  range  of 
Service  platforms.  Since  such  recommendations  should  not  be  made  based  on 
SIAP  performance  improvement  alone,  there  is  a  need  for  the  SIAP  SE  to  gain 
some  insight  into  more  operationally  oriented  MOEs  as  well.  Example  MOEs  are 
depicted  in  Table  3. 


Table  3.  Representative  IADS  MOEs 


Battlespace:  Location/Time  of  intercept  (engagement) 

Leakers:  Total  number  of  Hostile  weapon  systems  that  reached  their  ordnance 
release  points:  by  type 

Fratricide:  Total  Number  of  Friendly  targets  killed  by  Friendly  forces:  by  asset 
type  and  shooter  type 

Friendly  Attrition:  Total  number  of  Friendly  targets  killed:  by  asset  type 


However,  ‘final’  MOE/military  utility  analysis  is  the  purview  of  other 
operationally  oriented  organizations.  Therefore,  the  SIAP  SE  TF  must  coordinate 
its  analysis  and  recommendations  with  the  appropriate  warfighting  benefits 
assessment  organizations  within  the  DoD.  The  primary  interfaces  for  this 
coordination  are  JTAMDO  and  JFCOM.  (See  SIAP  Technical  Report  2001-002 
Measures  of  Effectiveness  (MOEs)  and  Measures  of  Performance  (MOPs)  for 
more  information.) 

The  SIAP  SE  is  working  closely  with  the  Director  of  JTAMDO  in  defining 
an  IADS  integrated  assessment  framework  for  evaluating  the  impact  on  military 
utility  of  system  level  improvement  recommendations.  Figure  5  depicts  the 
notional  roles  and  responsibilities  of  the  two  organizations.  Fundamentally,  the 
SIAP  SE  is  chartered  with  the  responsibility  of  evaluating  IADS  functional 
performance  and  associated  system  improvements  to  meet  SIAP  attribute 
metrics  requirements.  JTAMDO  is  responsible  for  evaluating  the  warfighting 
benefit  of  that  attribute  level  of  performance. 

While  tools  from  the  OITL  venue  such  as  VWC  and  Wargame  2000  fall 
under  the  leadership  purview  of  JTAMDO,  other  tools  span  the  gamut  of  IADS 
analysis.  To  support  the  federation  of  tools  required  to  link  system  improvements 
to  warfighting  benefit,  it  is  clear  that  the  SIAP  SE  must  work  closely  with 
JTAMDO  to  define  the  appropriate  linkages  between  pure  warfighting 
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assessment  venues  and  related  tools  and  other  venues/tools.  This  relationship 
requires  careful  coordination. 


Figure  5.  JTAMDO-SIAP  Relationships 

4.5  Lessons  Learned  Knowledge  Base  (LLKB) 

Current  CINC  guidance  and  memorandum  express  the  need  to  develop  a 
capability  to  “track  data  link  implementation  and  certification  across  all  members 
of  the  TAMD  FoS.”  Since  there  is  no  current  SIAP  focused  lessons  learned 
repository,  there  is  no  efficient  way  to  track  and  manage  MIL  STD  601 6A 
compliance.  This  results  in  the  same  SIAP  related  deficiencies  being  repeated 
year  after  year  at  various  live  exercises.  A  SIAP  LLKB  could  solve  these  issues 
by  tracking  military  standard  compliance  and  track  data  link  certification  across 
the  entire  SIAP  FoS  architecture,  and  providing  focus  for  SIAP  efforts  to  evaluate 
and  improve  warfighter  capability. 

The  purpose  of  building  a  LLKB  is  to  centralize  the  collection  of 
documented  assessments  from  observed  materiel  deficiencies.  By  pooling  this 
information  the  Services  can  address  the  impact,  frequency  of  occurrence,  and 
other  trend  data  to  help  focus  the  SIAP  SE  analysis  objectives.  The  LLKB  will 
enhance  warfighter  capability  evaluation  by  supporting  root  cause  analysis  of 
events  of  interest  gathered  from  selected  exercises,  HWIL,  and  OITL  events  of 
interest.  Additionally,  the  SIAP  LLKB  will  leverage  knowledge  from  previous 
activities  such  as:  JCIET  and  the  Joint  IADS  Working  Group  (JIADS  IWG);  other 
tests,  exercises  and  real-world  operations;  the  Joint  Composite  Tracking  Network 
(JCTN)  study  and  related  studies;  Joint  and  individual  Service  sensor  netting 
studies  and  analyses;  and  other  sources  of  lessons  learned.  The  information 
stored  in  the  knowledge  base  will  provide  a  basis  or  point  of  departure  for  future 
analyses  and  a  source  for  the  SIAP  Capabilities  and  Limitations  Document. 


Page  28 
9  July,  2002 


The  LLKB  will  provide  the  SIAP  SE  a  categorized  listing  of  TADIL  concerns 
and  deficiencies.  The  SIAP  LLKB  will  be  used  to  support  the  development  of 
issues  to  be  addressed  by  the  block  process. 


5.  Standard  Processes 

5.1  Block  Assessment  Methodology 

In  general,  the  following  analysis  methodologies  apply  to  each  Block  0 
through  n  item. 

5.1 .1  Use  All  Available  Data 

One  of  the  hallmarks  of  the  SIAP  SE  approach  is  leveraging  past  and 
ongoing  activities  in  which  SIAP  performance  assessments  may  be  conducted 
for  minimal  add-on  cost.  One  way  of  ensuring  that  ongoing  activities  are  able  to 
incorporated  into  the  SIAP  SE’s  overall  analysis  plan  is  for  the  SIAP  SE  to 
provide  assistance  in  improving  existing  data  collection  effort  and  verify  that 
sufficient  data  is  captured  to  calculate  the  standard  SIAP  attributes. 

Additionally,  by  documenting  data  from  events  as  they  occur  within  the 
LLKB,  the  SIAP  SE  provides  the  option  for  future  SIAP  assessments  to  conduct  a 
“re-analysis”  of  event  data.  This  analysis  might  provide  insight  into  a  different 
sector  of  IADS  performance,  and  is  can  only  be  possible  by  a  rigorous 
documentation  of  assumptions  and  limitations  of  specific  event  data. 

Another  valuable  source  of  data  that  can  be  leveraged  is  past  studies. 

Over  the  last  few  years,  there  have  been  many  assessments  of  SIAP-like  metrics 
in  many  different  venues  and  related  tools.  While  these  did  not  have  the  benefit 
of  calculation  of  the  standard  metrics  (which  did  not  exist  at  the  time),  many 
calculated  similar  metrics,  which  can  reasonably  be  expected  to  show  the  same 
trends  as  new  data  using  the  standard  calculations.  In  this  way,  past,  present, 
and  future  analyses  can  all  contribute  to  a  growing  body  of  knowledge  which,  if 
consistent,  can  increase  the  confidence  in  trends  shown,  and,  if  not  consistent, 
help  uncover  shortcomings  in  various  study  and  analysis  techniques  which  may 
need  improvement. 

5.1 .2  Establish  a  Standardized  Performance  Baseline 

At  the  highest  level,  the  SIAP  performance  baseline  used  to  determine 
when  the  SIAP  is  ‘good  enough’  will  be  determined  by  joint  warfighting 
requirements  as  evaluated  by  JTAMDO.  By  adding  the  recommended 
engineering  changes  of  each  SIAP  block  improvement  initiative  to  the  previous 
block  changes,  a  cumulative  record  of  progress  can  be  mapped  out,  with  the 
changes  shown  over  the  previous  block  performance.  An  increasing  absolute 
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value  of  the  metrics  then  culminates  in  a  set  of  values  for  the  last  block  upgrades 
that  meet  all  of  the  operational  requirements. 


To  establish  the  value  of  a  particular  proposed  engineering  level  change, 
or  block  of  changes,  on  the  SIAP,  assessment  tools  reveal  the  current  level  of 
SIAP  performance.  As  an  improvement(s)  is  introduced  changes  in  the  particular 
measured  parameter(s)  or  metric(s)  are  measured.  With  sufficient  control  over 
the  experimental  variables,  the  change  in  performance  of  the  introduced 
engineering  improvement(s)  can  be  determined.  This  type  of  comparative 
analysis  is  essential  to  the  SIAP  analysis. 

Quantified  Performance  Matrices  (QPMs)  will  be  used  to  track  changes  in 
metric  values  as  block  improvements  are  made  to  the  baseline  systems.  Tables 
4-7  provide  samples  of  QPMs  to  be  filled  out  as  block  analysis  is  conducted.  The 
first  three  tables  (4,  5,  and  6)  include  placeholders  for  attribute  and  MOE  values 
relative  to  each  CRS  used  for  block  analysis. 

Additionally,  the  SIAP  SE  plans  to  document  the  functional  capability  of 
different  systems  (as  part  of  a  functional  decomposition)  in  a  functional 
performance  matrix  similar  to  Table  7. 

Table  4.  QPM:  Air  Breather  Track  Attributes 


AIR  BREATHER  TRACK  (J3.2)  ATTRIBUTES 

CRS: 

NEA  III  2003 

AGCS  2010 

RT-2 

NEA  III  2010 

COMPLETENESS 

CLARITY 

Ambiguous  Tracks 

Spurious  Tracks 

CONTINUITY 

Characteristic  Track  Lifetime 

Lonqest  Track  Segment 

KINEMATIC  ACCURACY 

Position 

Velocity 

ID  COMPLETENESS 

ID  ACCURACY 

ID  CLARITY 

COMMONALITY 
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Table  5.  QPM:  Space  Track  Attributes 


SPACE  TRACK  (J3.6)  ATTRIBUTES 


_ _ CRS: 

COMPLETENESS  _ 

Track  Completeness _ 

LPE  Completeness _ 

IPP  Completeness _ 

CLARITY _ 

Ambiguous  Tracks _ 

Spurious  Tracks _ _ 

Ambiguous  LPEs _ 

Ambiguous  IPPs _ _ 

CONTINUITY _ 

Characteristic  T rack  Lifetime 

Longest  Track  Segment _ 

KINEMATIC  ACCURACY _ 

T rack  Position _ 

T rack  Velocity _ 

LPE  Position _ 

LPE  Time _ _ 

IPP  Position _ 

IPP  Time _ 

CORRECTNESS _ 


NEA  III  2003 


AGCS  2010 


RT-2 


NEA  III  2010 


I  Booster  Typing 

Post-Boost  Classification 
TIMELINESS _ 


Track  Initiation 

LPE  Delay 

Booster  Burnout  Estimate  Delay 

IPP  Time _ 

COMMONALITY _ 

Position/Time/Track  Number 

LPE _ 

IPP/Track  Number _ 
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Table  6.  QPM:  MOEs 


MEASURES  OF  EFFECTIVENESS 


CRS: 

NEA  III  2003 

AGCS  2010 

RT-2 

MEawm 

LEAKERS 

Total  #  of  hostile  weapon 
systems  that  reach  ordnance 
release  point,  by  type 

HOSTILE  ATTRITION 

Total  #  of  hostile  targets  killed, 
by  target  type 

FRIENDLY  ATTRITION 

Total  #  of  friendly  targets  killed, 
by  asset  type 

FRATRICIDE 

Total  #  of  friendly  targets  killed 
by  friendly  forces,  by  asset  type 
&  shooter  type 

WEAPON  EXPENDITURES 

Total  #  of  weapons  expended, 
by  type 

C2 

Total  #  of  engagements 
ordered,  by  type  &  target 

Blue  sortie  rates  ordered,  by 
mission,  force,  and  function 

BATTLESPACE 

Location/time  of  weapon 
commit 

Time/distance  from  initial 
detection  to  commit 

Location/time  of  intercept 
(engagement) 
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Table  7.  Functional  Performance  Matrix 


QUANTIFIED  PERFORMANCE  MATRIX 

SYSTEM  1 

SYSTEM  2 

SYSTEM  3 

TIME 

Time  Synchronization 

-  Difference  between  own  unit's  clock  time  &  UTC/USNO 

Element  data  exchange  latency  (definition  TBD) 

Time  to  qet  track  report  on  the  air 

-  Time  a  Dart.  obi.  meetinq  reportinq  crit.  has  a  valid  decl.  track  (track  init.  time) 

-  ReDortina  responsibility  declaration  for  each  track  at  each  evaluation  time 

\^m 

\m 

Lost  track  persistence 

i— 

I— 

Sensor  detection  ranqe 

Sensor  error 

Residual  Biases 

-  Sensor  aperture  ranqe  measurement  offset  bias 

-  Sensor  aperture  ranae  measurement  scale  factor  bias 

-  Sensor  aperture  ranoe-rate  measurement  offset  bias 

-  Sensor  aperture  bearinq  anqle  measurement  offset  bias 

-  Sensor  aperture  bearinq  anqle  measurement  offset  bias 

.  .Qoncnr  anortiire  bearinq  anale  measurement  scale  factor  bias 

1 1 II in i ii ii i 

-  Sensor  aperture  elevation  anqle  measurement  scale  factor  bias 

-  Sensor  aperture  roll  attitude/aperture  aliqnment  bias 

-  Sensor  aperture  pitch  attitude/aperture  aliqnment  offset  bias 

-  Rpnsnr  anerture  vaw  attitude/aperture  aliqnment  offset  bias 

-  TQ  as  a  function  of  time  for  each  track  _  _  __  _ 

TQ  consistency  _ _ _ __ _ 

DATA  CONNECTIVITY 

■1 

Time  distribution  of  connectivity  failures  (radio-to-radio) 

■ 

H 

Time  distribution  of  duration  of  connectivity  failures  (radio-to-radio) 

■ 

MBlliM 

■1 

Time  distribution  of  connectivity  failures  (track  store-to-  track  store) 

SISSl 

Time  distribution  of  duration  of  connectivity  failures  (track  store-to-track  store) 

■ 

DATA  REGISTRATION 

Geodetic  Registration 

Navigation  error(s) 

-  Difference  between  own  unit’s  navigation  measure  and  WGS-84 

-  Difference  between  inertial  navigation  measure  and  WGS-84 

Naviaation  Qpg  correctness 

-  Qdo  as  a  function  of  time  for  each  unit  reporting  Qpq 

■■Mil 

— 

— 1 

Navigation  Qpq  consistency 

_ 

_ 

— 

-  Qdo  as  a  function  of  time  for  each  unit  reportinq  Qpq 

mmmmm 

IU  Registration 

IU  registration  error  _ .  — 

IU  registration  error  covariance  consistency 

Sensor  Registration 

Sensor  registration  error  _  _ 

Sensor  registration  error  covariance  consistency 

Sensor  Gridlock 

Network-wide  absolute  sensor  gridlock  error 

Network-wide  absolute  sensor  qridlock  error  covariance  consistency 

■1 

Data  Processing 

ihi 

■HUP 

—1 

Computational  errors  ....  .  . 

CORRELATION/DECORRELATION 

Correct  correlation  rate 

-  Number  of  correct  correlations 

Correct  non-correlation  rate 

-  Number  of  correct  non-correlations 

Incorrect  non-correlation  rate 

-  Number  of  incorrect  (false)  correlations 

False  correlation  rate 

-  Number  of  incorrect  non-correlations  . . 

REPORTING  RESPONSIBILITY 

R2  correctness 

-  Reportinq  responsibility  declaration  for  each  track  at  each  evaluation  time 

COMBAT  ID 

ID  proqram  performance  ...  _  _ 

-  Svstem/iD  decl.  (friend,  hostile,  unk.,  etc)  on  each  track  at  each  eval.  time 

Gateoorv  proaram  performance  L 
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5.1 .3  Employ  Verification,  Validation,  and  Accreditation  (VV&A)  Techniques 

Two  important  questions  that  must  be  answered  by  the  SIAP  SE  are: 

1 .  How  much  fidelity  is  required  to  assess  improvements  to  the  SIAP? 

2.  How  much  confidence  does  the  SIAP  SE  have  in  assessment  results? 

Before  products  are  officially  endorsed,  the  SIAP  SE,  the  SIAP  analysis 
Accreditation  Authority,  ensures  that  M&S  analysis  has  gone  through  the 
appropriate  level  of  VV&A.  The  SIAP  SE  will  follow  DoD  5000  guidance  for 
VV&A  efforts  of  all  SIAP  related  assessments. 

The  purpose  of  VV&A  is  to  assure  development  of  correct  and  valid 
evaluations  from  M&S  analysis  efforts.  VV&A  processes  are  performed  to 
establish  the  credibility  of  the  models  and  simulations  used  in  analysis 
applications.  Credibility  depends  on  model  simulation  approximations  -  not  in  an 
absolute  sense,  but  relative  to  the  model  approximations  needed  for  the  specific 
application.  Hence,  SIAP  needs  correct  network  approximations  in  network 
simulation  to  determine  how  a  particular  JDN  fix  will  affect  the  SIAP,  and  a  good 
approximation  in  system  effectiveness  simulations  to  determine  how  a  particular 
improvement  in  the  SIAP  will  impact  a  system's  performance.  The  decision  on 
whether  or  not  a  simulation  provides  the  necessary  degree  of  accuracy  depends 
not  only  upon  the  inherent  characteristics  of  the  simulation,  but  also  upon  how 
the  simulation  will  be  used,  and  upon  the  significance  of  any  decisions  that  may 
be  reached  on  the  basis  of  the  simulation’s  outputs. 

Technical  or  resource  limitations  may  place  limitations  on  VV&A  activity, 
requiring  related  processes  to  be  tailored  in  a  way  that  is  less  than  ideal  from  a 
formal  VV&A  perspective.  The  Accreditation  Authority,  in  this  case  the  SIAP  SE, 
must  take  any  such  limitations  into  account  when  reaching  a  conclusion  for  the 
approval  or  disapproval  of  the  use  of  a  simulation  analysis. 

The  specifics  of  VV&A  for  each  of  the  models  and  simulations,  and  V&V 
methodologies  used  in  each  block  will  be  addressed  in  the  associated  block 
appendix. 

5.1 .4  Standardized  Data  Management  and  Analysis  Plan  (DMAP) 

The  purpose  of  the  standardized  DMAP  is  to  provide  a  single  source  for 
supporting  SIAP  analysis  with  regard  to  live  exercises  or  events.  The 
standardized  DMAP  conveys  the  following: 

1.  Provide  a  description  of  the  critical  experiments  to  be  conducted  to  evaluate 
SIAP  systems 

2.  Explain  how  the  SAT  will  use  the  data  collected  at  events  to  assess  Joint 
IADS  performance 
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3.  Provide  guidelines  on  what  data  needs  to  be  collected  to  support  the  analysis 
efforts. 

4.  Describe  how  the  SAT  will  evaluate  progress  in  the  ability  to  build  and 
maintain  a  SIAP. 

5.  Lay  down  a  schedule  that  outlines  roles  and  responsibilities  of  participants 
(SAT,  Services,  test  staff,  etc)  before,  during,  and  after  the  event 

6.  Explain  the  importance  and  utility  of  a  lessons  learned  knowledge  base 

7.  Provide  a  streamlined  method  for  event  directors  and  planners  to  provide 
event  details 

The  guidelines  in  the  standardized  DMAP  will  evolve  as  events  are  conducted 
and  analysis  of  IADS  performance  evaluated. 

The  standardized  DMAP  also  documents  critical  experiments  that  the  SAT 
will  focus  on.  In  general,  a  critical  experiment  is  designed  to  address  a  specific 
SIAP  issue  or  concern.  Each  experiment  has  associated  measurements  and  a 
specified  analysis  method.  Ideally,  all  of  these  experiments  would  be  carried  out 
in  each  event,  but  there  may  be  venue-specific,  tool-specific,  or  event-driven 
limitations  which  may  require  restriction  to  some  subset  of  the  critical 
experiments  provided,  or  slight  modifications  of  specific  experiments.  Any  such 
limitations  will  be  specified  in  the  DMAP  (if  anticipated),  or  in  the  data  analysis 
reports  (if  not),  for  the  particular  venue,  tool,  or  event  in  question. 

Critical  Experiments: 

1 .  Time  Synchronization 

2.  Sensor  Tracking/Reporting  Accuracy 

3.  Data  Registration 

4.  Automatic  Local-to- Remote  Track  Correlation/Decorrelation 

5.  Identification  Processing 

6.  Formation  Tracking  and  Assessment 

7.  Model  and  Simulation/Stimulation  Fidelity 

8.  Commonality 

9.  Precise  Participant  Location  and  Identification  (PPLI)  Accuracy 

1 0.  Multi-Link  T  ranslation/Forwarding 
1 1  .TBMD  Performance 

12.  Early  Warning  (EW)  Performance 

Figure  6  below  displays  how  DMAPs  are  integrated  into  the  SIAP  SE’s  M&S 
process. 
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5.1 .5  Documentation  of  Findings:  Technical  Reports 

The  SIAP  SE  will  document,  in  the  form  of  technical  reports,  all  of  its 
analysis  results  as  well  as  major  decisions  made  in  the  course  of  analysis 
(definitions,  goals,  assessment  guidelines,  etc.).  Some  areas  may  require 
several  reports,  or  a  report  that  is  frequently  updated. 

To  ensure  full  Service  participation  and  joint  consensus,  all  SIAP  SE 
technical  reports  are  subjected  to  an  intensive  Service/Agency  review  and  editing 
process  prior  to  final  approval  and  publication.  This  process  must  include  at 
least  the  following  stages  of  input  and  revision. 

•  First,  the  appropriate  SIAP  working  group  meets  to  address  the  major 
issues  and  to  note  the  views  and  concerns  of  all  members. 

.  Then  a  draft  document  is  drawn  up  within  the  working  group  and 

circulated  (usually  electronically)  among  the  membership  for  discussion 
and  revision.  The  document  is  revised  until  consensus  is  attained  within 
the  working  group. 

.  Following  (or  simultaneous  with)  this  stage  of  revision,  the  draft  report  is 
reviewed  by  the  SIAP  SE  TF  command  chain  for  consistency  with  the 
SE’s  objectives  concerning  the  problem  area  under  study,  possibly 
resulting  in  further  revision. 

.  Next,  the  revised  draft  is  released  to  a  much  larger  body  of  POCs  in 
various  offices  throughout  DoD,  including  OSD,  the  S/A  designated 
representatives  of  each  Service  and  joint  Agencies,  including,  but  not 
limited  to,  those  represented  on  the  working  group.  An  attempt  is  made 
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,  nr)n  contaot  with  a  reasonable  claim  to  having  both  an 

Interest  inTne  anal/hs^^  ,0 

.  All  feedback  from  these  ^ 's  f "  d [,0  when  not  incorporated 

The""  z^d  untconsensPus  is  attained  at  least  among  the 

Sendee  and^tesignated  A=P"Sis  then  sent  through  the 

•  SIAI^SE  TF^command^eview  chahTagafn  for  ®  J'nal  check,  and 

classification  permitting,  distributed. 

„  should  be  noted  that  the 

the  Agencies  represented  on  the >  working  g  ^  PP^  a(  me  stage  of  more 

formal  Servlce/Ageni^  approval  prior  to  submission  to  the  SIAP  SE. 

several others  arT in  progress  as  ^cmple^edmfts^Tvarious  staged  oHhe'r©^©w 
process. 

5.2  Integration  of  Multiple  Venues 

As  stated  previously  no  °"®f 

,o-end  system  to  m™a,V  IAP  lays  out  a  strategy 

has  its  own  strengths  and  wf  ak"e^  „ue  t0  compensate  for  the  weaknesses 

of^rthers  S  The^esutt  oSstmtegy  should  be  a  set  of  SIAP  assessment  results, 

which  is  better  than  the  individual  sum  of  the  parts. 

Table  8  beiow  ^ 

assessment  venue  available  to  the  SIA  ■  9  venue  to  overcome 

with  the  tools  available  for  each  assessment  effort. 

5.2.1  Synergy 

Table  8  illustrates  some  of ^T^astssme"6'  °' 
fidelity  in  the  integrated  assa®s["f"ab|g  £9  relative  fidelity  of  the  representations 
capable  of  providing  the  .  .  ’  j  subjectively  ranked  with  respect  to 

sssssrisss  i. ™ . 

scale  is  1-4,  with  4  being  best. 
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Table  8.  Venue  Ranking 


jsj  „  >.  i  ^  %  ^  \ 

'^Hardware  inthe  loop/. 

iDiditarsfmsJ,;. 

■t>^Diaital  Sirris^M. 

Netwoikl^fc 

Performance; 

-  1  (EADTB...)  • 

■BHAh! 

Warfighter 

Benefits 

4 

3 

2 

4 

[ft  of 

Participants 

1 

1 

4 

4 

fEvent  Cost" 

3 

1 

4 

3 

'Tool  Cost 

3 

4 

3  1 

2 

Excursions 

1 

2  j 

4 

4 

Control  of 
Variables 

3 

1 

4 

4 

Network/Link 

2 

1  I 

4 

2 

;  Terminal 

3 

3 

3 

3 

Host 

;  Processing 

4 

4 

2 

3 

Sensor 

4 

4 

2 

3 

Human 

;  Behavior®^?9  j 

4 

3 

2 

2 

The  ratings  against  the  features  in  Table  8  show  that  most  of  the  important 
assessment  characteristics  desired  are  covered  quite  well  in  one  venue  or 
another.  Therefore,  if  reasonable  linkages  can  be  created  such  that  data 
produced  in  a  venue  that  is  strong  in  certain  characteristics  can  be  used  to 
compensate  for  weaknesses  in  other  venues,  then  it  is  possible  to  create  a  more 
effective  combination  of  capabilities  than  any  one  venue  by  itself.  This  will  create 
a  path  from  technical  MOPs,  to  SIAP  attributes,  and  on  to  warfighting  MOEs. 

The  number  of  features  requiring  theater  level  analysis  also  indicates  the 
necessity  for  tools  capable  of  SIAP  performance  assessment  at  that  level. 

The  initial  thrust  of  the  SIAP  SE’s  efforts,  as  directed  by  the  JROC,  is 
towards  improving  the  JDN,  with  initial  emphasis  on  Link  16.  This  direction 
dictates  a  need  for  high  fidelity  data  link  and  network  models,  and  high  fidelity 
modeling  of  specific  host  functionality  directly  associated  with  block  changes  to 
support  the  required  analysis. 

5.2.2  Linkages 

The  frames  in  Figure  7  illustrate,  at  a  very  high  level,  the  process  that  will 
lead  to  the  most  effective  combination  of  the  strengths  of  the  various  modeling 
and  assessment  venues  included  in  the  IAP. 
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Exercise/Engineering 

^  /Data  Driven -  Analysis 

rrxl — , — A— ^  AFTER 

Outputs:  Attributes 

1 

A  \  f\  iew/  Block  0 

/fVV  \  before 

T  v  V  '  iew/o  Block  0 

Noise 

Validate  Digital  Models 

-  "  c  ivL 

/ 

BgSBP 

Before/after  changes  made  tfr 
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Figure  7.  IAP  Venue  Linkages 


Exercise/Engineering  ‘Data  Driven’  Analysis 

Live  and  HWIL  events  (e.g.,  JDEP,  JCIET,  and  Roving  Sands)  provide 
extensive  data  for  quick  assessments  of  SIAP  measures.  Systems  and 
subsystems  that  can  influence  the  SIAP  do  not  have  to  be  approximated, 
because  (many  of  the)  real  systems  are  used.  The  Services  and  joint  analysis 
communities  have  developed  a  number  of  tools  with  which  to  process  and 
analyze  real-world  data  collected  at  such  venues. 

Depending  on  the  configuration  of  platforms  participating  in  these  venues, 
it  may  be  possible  to  collect  and  analyze  various  SIAP  metrics  before  or  after 
SIAP  improvements  have  been  implemented  in  hardware  and  software.  The 
ideal  situation  would  be  to  run  each  test  with  each  participating  platform  first  in 
the  before  configuration  against  some  standard  scenarios  and  vignettes,  collect 
data,  and  then  calculate  SIAP  attributes.  Next  run  the  same  test  over  with  the 
same  platforms  and  configurations,  except  for  the  addition  of  changes.  In  so  far 
as  the  test  conditions,  configurations,  scenarios,  operator  actions,  hardware  and 
software  status,  and  vignettes  can  be  held  constant,  the  comparison  between 


Page  39 
9  July,  2002 


those  data  sets  should  reveal  the  changes  in  SIAP  metrics  due  to  the  system 
changes  introduced. 

Unfortunately  a  variety  of  constraints  and  variables  prevent  the  absolute 
control  of  open-air  analyses,  and  full  theater  level  analysis  is  not  practical. 
Weather,  hardware  failures,  system  availability,  and  operator  training  are  a  few  of 
the  variables  that  will  change  from  event  to  event.  Consequently,  before  and 
after  comparisons  between  different  exercises  may  not  be  statistically  significant. 
There  are  too  few  platforms,  too  many  uncontrollable  variables,  and  essentially 
no  guaranteed  repeatability. 

In  an  attempt  to  overcome  some  of  these  limitations,  the  Services  and  the 
joint  community  are  pursuing  several  initiatives.  One  of  the  goals  of  JDEP  is  to 
be  able  to  re-run  scenarios  with  identical  hardware  in  the  loop,  except  for  certain 
specific  changes  that  are  made  for  evaluation  purposes.  This  approach 
overcomes  many  of  the  limitations  of  the  open  air  exercise  venues  cited  above, 
but  it  does  not  solve  the  problem  of  too  few  platforms,  and  difficulty  in  evaluating 
performance  with  varying  numbers  of  platforms  with  and  without  changes,  etc.  In 
addition,  to  evaluate  post-block  changes  performance,  desired  changes  must  be 
implemented  in  actual  systems.  This  can  be  a  costly  and  time-consuming 
endeavor. 

Another  approach,  data  perturbation  analysis,  has  been  pursued  by  CNA 
as  a  means  for  getting  more  out  of  the  reconstruction  of  live  events.  By  using 
live  system  data  directly,  this  approach  accounts  for  the  specific  software  and 
hardware  configurations  actually  used  by  platforms  in  the  exercise.  SIAP  metrics 
can  then  be  calculated  using  recorded  data  generated  from  those  configurations. 
These  capabilities  have  come  to  be  called  ‘data  driven’  models. 

By  changing  the  method  by  which  the  actual  exercise  data  is  processed 
(e.g.,  running  the  data  through  the  new  Block  0  correlation  algorithms)  at  each 
participating  platform,  the  data  driven  models  can  measure  the  effects  of  different 
processing  schemes  against  the  recorded  data  baseline.  Except  for  the  new 
processing  logic  introduced,  this  approach  has  the  advantage  of  still  being  based 
on  the  actual  hardware  and  software  configurations  of  real  platforms,  and  real 
world  recorded  data.  However,  it  is  still  hampered  by  the  limitations  of  few 
platforms,  sparse  data,  and  an  approximation  of  what  actual  platform 
implementations  of  various  SIAP  changes  might  look  like. 

Digital  Simulation  Calibration/Validation 

Modeling  and  simulation  capabilities  in  this  realm  allow  scale  up  to  the 
theater  level  performance  evaluation  that  is  the  goal  of  SIAP  analysis.  It  also 
eliminates  many  of  the  other  shortcomings  associated  with  exercise  and  live-fly 
events  (e.g.,  weather,  hardware  failures,  operator  training,  etc.).  However,  digital 
simulations  must  create  representations  of  how  all  of  the  primary  and  contributing 
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systems  work  (in  lieu  of  having  actual  HWIL  and  OITL  capability)  which  provide 
data  to  the  SIAP  functions.  The  fidelity  of  system  representations  will  not  be  as 
good  as  with  the  HWIL  or  exercise  venues.  Depending  on  the  specific  type  and 
purpose  of  a  particular  analysis,  many  of  the  subsystems  do  not  have  to  be  of 
perfect  fidelity  since  their  role  with  respect  to  the  primary  analysis  is  secondary. 
Figure  8  is  a  notional  example  of  this  concept  assuming  that  network  issues  are 
the  primary  analysis  objective. 
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Figure  8.  Notional  M&S  Fidelity  Requirements 
for  Network  Issues 


Due  to  the  complex  integration  required  between  host  systems  and  the 
data  link,  the  problem  of  deciding  which  host  functions  must  be  modeled  in  detail 
and  which  can  be  approximated  with  a  lower  fidelity  implementation  is  difficult. 

At  a  minimum,  the  system  functions  that  are  under  evaluation  (i.e.,  the  ‘primary’ 
functions)  have  to  be  modeled  at  high  fidelity  to  assess  both  before  and  after 
implementation  performance  within  a  specific  digital  simulation  environment. 

One  of  the  significant  differences  between  tools  in  the  digital  simulation 
venue  and  those  in  the  live,  HWIL,  OITL  venues  is  that  it  is  more  difficult  to 
model  the  before  case  in  a  digital  simulation  than  it  is  to  model  the  after  case. 
This  fact  is  primarily  driven  by  difficulties  in  acquiring  the  necessary  information 
to  model  the  before  case,  as  opposed  to  the  inability  to  model  the  required 
functionality.  When  implementing  a  SIAP  change  in  a  digital  simulation,  it  will  be 
clearly  delineated  which  platforms  will  implement  function-changes  for  particular 
analysis  cases.  However,  it  is  not  as  clear  how  platforms  perform  functions  in 
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the  before  case  because  it  is  not  always  known  or  documented.  Additionally,  in 
the  after  case,  the  model  developer,  in  consultation  with  appropriate  SMEs,  is 
free  to  choose  the  method  of  integration.  As  with  data-driven  tools,  this  after 
case  is  only  an  approximation  of  what  actual  platform  implementations  of  various 
SIAP  changes  might  look  like. 

Modeling  the  before  case  requires  a  detailed  knowledge  of  how  each 
platform  actually  does  a  specific  function,  followed  by  a  translation  of  that 
scheme  into  modeling  software.  Program  office  review  of  the  implementation  is 
then  required  for  validation  of  the  model  representation.  This  process  requires 
much  work  and  time.  It  is  our  experience  that  this  is  a  very  tedious  and 
challenging  task.  Program  Managers  are  often  not  very  forthcoming  with  system 
performance  data  that  might  show  adverse  system  performance.  Even  when 
completed,  the  implementations  in  model  software  will  still  only  approximate  the 
performance  of  the  real  systems.  This  is  why,  in  Table  8,  the  digital/constructive 
simulation  baseline  functions  listed  as  sensor  and  host  are  rated  lower  than  in 
the  live  and  HWIL  venues,  though  some  digital  models  are  better  than  others. 

One  of  the  simplifications  usually  made  in  the  digital  modeling  realm  is 
that  the  lower  fidelity  modeling  of  the  secondary  influences  is  done  assuming  that 
those  functions  work  correctly  and  consistently  across  all  platforms.  With  such 
an  idealistic  assumption,  the  after  SIAP  performance  values  calculated  in  the 
digital  simulations  may  be  better  or  worse  than  they  should  be.  However,  if  the 
absolute  values  of  SIAP  attributes  derived  from  the  ‘data  driven’  tools  are  used  to 
help  calibrate  the  output  of  the  digital  simulations  under  similar  scenario 
conditions,  there  will  be  more  confidence  that  the  absolute  metric  values 
produced  by  the  digital  simulations  are  more  accurate.  At  the  same  time  there 
will  be  traceability  back  to  the  real-world  events  where  no  assumptions  had  to  be 
made.  The  data  driven  and  digital  simulation  modelers  will  be  tasked  to  work  with 
each  other  to  develop  the  best  interfaces  between  the  two  venues  to  support  this 
synergy. 

Map  through  Theater  Architecture 

To  determine  the  significance  of  any  particular  engineering  change  on 
warfighting  MOEs,  it  must  be  placed  in  the  overall  operational  context  in  which  it 
will  be  used.  Ideally  the  changes  in  attributes  caused  by  engineering  changes 
will  flow  in  an  automated  way  into  the  mission  level  model  from  a  higher  fidelity 
model,  or  a  model  with  end-to-end  capability  may  be  used.  This  process  provides 
a  trace  from  exercise  data  through  calibrated  attributes  to  engagement  MOEs. 
However,  today  an  integrated  end-to-end  analysis  capability  does  not  exist. 
Therefore,  initially,  for  many  specific  block  changes,  separate  high  fidelity  M&S 
tools  will  be  required  for  MOP  level  evaluations.  Then  the  higher  theater  level, 
SIAP  attribute  tools  will  need  to  be  adapted  to  reflect  the  effects  of  MOP 
improvements  on  the  SIAP  (and  selected  MOEs). 
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Engagement/Campaign  Models 

The  lower  right  quadrant  of  Figure  7  represents  an  attempt  at  an  even 
greater  level  of  roll-up  to  higher  level  engagement  and  campaign  MOEs.  For 
example,  an  MOE  model  can  measure  the  number  of  weapons  expended  in  any 
given  scenario,  and  might  show  that  a  strategy  of  multiple  shots  at  every  target 
results  in  the  highest  number  of  kills.  However,  if  the  weapon  supply  and 
platform  loadouts  are  considered,  such  a  strategy  may  prematurely  deplete 
weapons  supplies,  leaving  nothing  for  later  targets  resulting  in  more  leakers,  loss 
of  Blue  Force  defended  assets,  etc.  Still  other  models  are  designed  to  assess 
high-level  campaign  metrics  such  as  days  required  to  win,  etc.  These  higher 
level  engagement  and  campaign  models  will  use  results  from  the  lower  levels  as 
inputs. 


The  trace  from  real-world  data  to  campaign  level  MOEs  can  be  completed 
following  the  path  shown  in  Figure  7.  While  the  quadrants  display  the  notional 
linkages  between  levels,  the  many  details  of  the  interfaces  between  levels  need 
to  be  determined,  and  may  vary  from  model-to-model,  and  as  models  evolve. 
These  details  need  to  be  worked  out  individually  for  each  block.  Therefore,  they 
will  be  addressed  in  each  specific  block  appendix  to  this  document. 


6.  Summary 

The  SIAP  SE  was  chartered  to  implement  a  disciplined  system  engineering 
process  for  the  purpose  of  evaluating  SIAP  IADS  shortfalls,  and  recommending 
improvements  to  joint  warfighting  effectiveness.  To  accomplish  the  task  at  hand, 
the  SIAP  SE  must  establish  a  standardized  analytical  infrastructure  composed  of 
teams,  tools,  and  processes.  This  infrastructure  is  supported  by  the  Services 
and  Joint  Agencies  and  leverages  the  significant  DoD  investment  in  evaluation 
capabilities.  The  SIAP  SE  will  use  this  infrastructure  to  support  a  block 
improvements  process  aimed  at  incrementally  building  an  objective  SIAP. 

The  coordination  of  the  various  contributors  who  support  the  teams,  venues 
and  processes  is  a  daunting  task.  To  bring  order  to  the  chaos,  the  SIAP  SE  has 
developed  this  Integrated  Assessment  Plan,  providing  an  overarching 
perspective  for  all  SIAP  analysis.  This  IAP  leverages  and  integrates  the  results 
of  many  past  and  ongoing  assessment  efforts.  It  establishes  standardized 
metrics,  scenarios,  and  analysis  methods  to  use  across  venues  for  the  purpose 
of  comparing  and  contrasting  results  and  standardizing  analysis.  It  also  provides 
for  development  and  upgrade  of  tools  over  time,  including  VV&A  of  all  tools  used 
in  SIAP  analysis. 

The  SIAP  assessment  approach  is  designed  to  leverage  and  synergize 
results  of  multiple  analytical  methodologies,  with  the  aim  of  providing  the  highest 
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quality  recommendations  to  the  JROC  within  the  time  and  budgetary  constraints 
imposed. 
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Integrated  Assessment  Plan 

Identification 

Institute  of  Electrical  and  Electronics  Engineers 

Integrating  Integrated  Product  Team 

Infrared 

Interface  Unit 

Integrated  Working  Group 

Joint  Combat  Identification  Evaluation  Team 
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Missile  Defense  Agency 
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TQ 
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WG 
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Quantified  Performance  Matrix 
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Verification,  Validation,  and  Accreditation 
Virtual  Warfare  Center 
Working  Group 

Working  Integrated  Product  Team 
Weather  Effects 
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APPENDIX  A:  Tool  Descriptions 


Introduction 

The  purpose  of  Appendix  A  is  to  give  a  high  level  description  of  known  models  and  tools 
available  for  SIAP-related  analyses.  The  information  contained  in  this  appendix  is  a  result 
of/based  on  inputs  from  model  POCs,  users,  and  official  model  overview  documents.  This 
appendix  is  a  living  document  and  will  be  updated  to  recognize  new  capabilities  and  tools  as 
they  emerge. 

Each  description  contains  the  following  sections: 

1 .  Overview  -  What  does  it  do,  for  what  purpose  was  it  created,  what  platforms  does  it  run 
on? 

2.  Design  -  How  does  it  work? 

3.  Assumptions  and  Limitations  -  What  assumptions  and  generalizations  does  the  user 
have  to  consider  when  interpreting  the  results? 

4.  SIAP  Contribution  -  What  problems  does  it  solve  for  the  SIAP  world? 

The  descriptions  provided  in  this  appendix  provide  an  indication  of  how  the  models/tools 
work  and  how  they  might  fit  into  a  specific  analysis  process.  For  a  more  detailed 
description  of  their  precise  uses  in  previous  Block  n  analysis,  see  the  appropriate  appendix. 

The  following  is  a  list  of  the  models/tools  described  in  this  document  and  their  location: 
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A.1  Air  Defense  Simulation  (ADSIM) 

A.  1.1  Overview 

ADSIM  provides  detailed  tactical  data  link  modeling,  including  realistic  sensor,  tracker 
and  host  system  function  representations.  The  detailed  sensor  and  tracker  models 
provide  the  basis  for  a  realistic  local  track  file  that  is  correlated  with  a  realistic  network 
track  file.  ADSIM  then  models  how  host  systems  turn  selected  local  tracks  into  Tactical 
Digital  Information  Link  (TADIL)  J  messages  and  transmit  them  within  a  theater  of 
operations.  ADSIM  models  host  systems  properly  calculating  track  quality  (TQ), 
observing  reporting  responsibility  (R2)  rules,  metering  the  messages,  and  buffering  the 
messages,  while  also  modeling  JTIDS/MIDS  terminals  utilizing  packing  limits,  directly 
importing  real  (operational)  network  design  constraints,  then  transmitting  the  messages 
through  the  Link-16  network. 

A.1. 2  Design 

The  communications  model  is  capable  of  simulating  the  effects  of  jamming  and  the 
impacts  of  limited  time  slot  availability  on  track  update  rates  as  well  as  the  number  of 
tracks  on  which  R2  can  be  maintained  in  a  Joint  Theater  Air  and  Missile  Defense 
(JTAMD)  environment.  MITRE  has  developed  complementary,  detailed  post¬ 
processing  tools  to  facilitate  analysis  and  graphically  display  the  results  of  the 
simulation.  The  Link-16  portion  of  the  communications  model  is  the  only  JTIDS 
Program  Office  endorsed  model  for  Link-16  loading  studies.  BMDO  and  JTAMDO  have 
funded  the  MASC’s  use  of  ADSIM  on  numerous  occasions  to  assess  a  wide  variety  of 
Link  1 6  performance  issues. 

ADSIM  directly  utilizes  output  from  the  Automated  Terminal  Initialization  (ATI)  tool  to 
develop  actual  Link  1 6  network  designs,  based  on  significant  interaction  with  Service 
and  Joint  representatives.  The  ATI  is  a  compatible  prototype  of  the  Joint  Link  16 
network  design  tool  that  is  used  by  all  of  the  Service  Network  Design  Labs.  MITRE  has 
developed  a  complementary  tool  to  automatically  import  ATI-developed  Link  16  network 
designs  directly  into  ADSIM,  which  properly  uses  all  of  the  information  provided. 

As  an  analysis  provider  for  the  Phase  2  JCTN  Study,  MITRE  also  has  developed  the 
capability  to  model  composite  tracking  as  defined  for  JCTN  within  ADSIM.  Composite 
tracking  refers  to  a  process  by  which  sensor  measurements  from  multiple  sensor 
platforms  are  combined  to  form  a  single  track  (ideally)  for  each  target  observed. 

ADSIM  has  demonstrated  its  ability  to  represent  JCTN  performance.  CEC  is  not  the 
same  as  JCTN.  Representing  all  of  the  accommodations  that  the  CEC  program  had  to 
make  in  order  to  integrate  data  fusion  into  the  CEC  network  has  necessarily  been 
approximated. 

ADSIM  is  designed  to  support  the  simulation  of  composite  tracking  through  the  use  of 
its  sensor  objects  and  command  center  objects.  Sensor  objects  are  used  to  simulate 
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the  operation  of  the  various  sensor  systems  deployed.  They  detect  targets  and  provide 
sensor  measurements  to  designated  command  centers  where  the  air  and  space  tracks 
are  established,  updated  and  reported  via  Link  16.  Various  combinations  of  sensor 
objects  and  command  center  objects  may  be  implemented.  For  instance,  to  model 
each  sensor  as  an  individual  JTIDS  Unit  (JU)  wherein  the  individual  JU  prepares  and 
reports  tracks  using  only  its  own  sensor  measurements,  a  separate  command  center 
may  be  established  for  each  sensor.  To  model  composite  tracking,  a  single  command 
center  can  be  established  to  process  the  sensor  measurements  from  several 
contributing  sensors. 


A.2  Analysis  in  Motion  (AIM) 

A^MA.2.1  Overview 

CSCI's  tool  suite  accounts  for  dynamic  interactions  between  platforms,  sensors  and 
networks.  CSCI’s  principal  theater  air  and  missile  defense  model  is  a  detection  through 
kill  simulation.  In  this  model,  measurements  created  by  sensors  in  a  TAMD  architecture 
are  subjected  to  gating  and  correlation  procedures  to  associate  measurements  with 
tracks.  Associated  measurements  are  used  to  update  a  Kalman  tracking  filter  for  real 
and  false  tracks.  Measurements  which  fail  to  associate  with  any  track  spawn  new 
tentative  tracks.  Subsequent  measurements  can  be  associated  with  tentative  tracks 
and  the  tentative  tracks  can  be  promoted  to  confirmed  or  network  reportable  track 
status.  The  accuracy  of  measurements  and  the  track  update  rate  determine  the  error 
around  a  specific  track  and  influence  the  size  of  the  adaptive  gate  around  this  track. 
Therefore,  network  performance  directly  influences  the  accuracy  of  tracks  which 
influences  whether  tracks  tend  to  dual  or  swap.  The  AIM™  model  can  model 
measurement  sharing  and  reporting  responsibility  networks  either  individually  or 
simultaneously  in  a  single  run.  This  modeling  approach  allows  us  to  identify  instances 
of  interest  along  with  the  preceding  history  and  conditions  leading  to  the  interval  of 
interest.  In  other  words,  if  a  tactical  situation  involving  several  sensors  and  targets 
leads  to  a  merge,  dual  or  swap  situation;  we  can  replicate  the  events  preceding  the 
event,  define  explicitly  why  it  occurred  and  what  means  are  available  to  resolve  the 
undesirable  air  picture  situation. 

AFk2A.2.2  Design 

Automatic  local-to-remote  track  correlation/decorrelation  implies  a  relative  performance 
assessment  of  techniques  for  avoiding  erroneous  network  reporting  of  dual,  merged  or 
swapped  track  reports.  As  stated  earlier,  our  capability  applies  directly  to  the  creation 
of  the  erroneous  reports  and  our  approach  towards  representing  the 
association/correlation  processes  gives  us  the  flexibility  to  evaluate  the  various 
approaches  towards  resolving  track  ambiguities  by  explicit  representation  of 
measurement,  gating,  correlation  and  tracking  processes.  Our  approach  lends  itself  to 
robustness  and  sensitivity  analyses  appropriate  to  this  topic  in  that  we  can  define  the 
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number  of  participating  sensors  and  corresponding  detection  measurement  properties, 
the  duration  of  the  ambiguous  interval  and  the  effectiveness  of  processes  for  resolving 
the  ambiguity.  This  might  be  best  accomplished  by  first  representing  various  network 
alternatives  in  models  which  assess  network  delays  and  capabilities  directly  and  then 
calibrating  one  or  more  networks  in  an  AIMTM  analysis  from  these  network  specific 
models.  AIM™  would  then  be  used  to  generalize  the  network  performance  model 
findings  to  force-on-force  level  analysis  and  would  report  traditional  operational  metrics. 
The  CSCI  team  is  experienced  in  the  use  of  detailed  sensor  and  network  models  to 
calibrate  AIM™  and  extend  results  to  force-on-force  level  analyses. 

ID  conflict  resolution  rules  implies  an  assessment  of  such  rules  in  the  presence  of  an 
unambiguous  tracking  situation.  Our  capability  allows  for  direct  implementation  and 
evaluation  of  these  rules  and  procedures  in  increasingly  more  stressing-detection  and 
tracking  environments.  This  provides  for  the  assessment  of  the  robustness,  frequency 
and  duration  of  occurrence,  precursor  conditions,  and  process  intervals.  That  is,  we  can 
measure  the  frequency  of  occurrence  of  ID  conflicts  and  the  duration  of  time  during 
which  the  conflicts  exists  -  explicitly  instead  of  statistically.  We  track  the  precursor 
conditions  leading  to  the  ID  conflict  and  the  time  it  takes  to  remove  the  conflict.  We 
need  to  understand  the  relationship  between  this  ICP  and  the  ICP  related  to  taxonomy 
and  symbology  to  clearly  define  assessment  goals. 

Formation  tracking/correlation  implies  techniques  for  distinguishing  resolution 
differences  between  sensors  and  how  to  manage  the  reporting  of  tracks  on  surveillance 
and  sub-nets  on  Link-16.  We  explicitly  represent  multiple  networks,  the  interactions 
between  them  with  different  combinations  of  participating  sensors  on  each  network. 


A.3  Automated  Reconstruction  Correlation  Tool  for  Interoperability 
Characterization  (ARCTIC) 

AAMA.3.1  Overview 

The  ARCTIC  tool  was  developed  at  CNA  to  aid  in  rapid  reconstruction  of  air  defense 
exercises  with  the  purpose  of  interoperability  analysis  in  mind.  ARCTIC  automates  the 
process  of  matching  radar  track  data  with  ground  truth  data  (usually  obtained  from  GPS 
pods  on  participating  aircraft)  recorded  during  the  event.  The  track  matching  process 
uses  simple  kinematics  rules  and  hysteresis  to  choose  the  “best”  match. 

AAk3A.3.2  Design 

ARCTIC  users  can  control  rules  used  to  generate  matches  including  kinematics 
correlation  windows  as  well  as  how  much  hysteresis  is  applied  to  maintain  track 
matching  continuity  on  objects  within  a  formation.  These  options  allow  the  user  to  tailor 
the  reconstruction  for  suitability  with  certain  kinds  of  metrics. 
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ARCTIC  is  fully  compatible  with  the  Performance  Evaluation  Tool  (PET)  for  generating, 
viewing,  and  graphically  editing  reconstructions.  It  is  also  compatible  with  the  standard 
WAM  data  format,  such  as  used  in  ASCIET  /  JCIET  events. 

ARCTIC  was  benchmarked  against  ASCEIT  99  manual  reconstructions  and  was  used 
extensively  for  analyses  of  ASCEIT  00  events.  The  ARCTIC-PET  combination  has 
been  employed  for  reconstruction  of  Underway  Events  10,  11,  and  12  (TechEval)  for 
the  Cooperative  Engagement  Capability  (CEC)  preparation  for  OPEVAL. 


A.4  Extended  Air  Defense  Simulation  (EADSIM)  Enhancements 
At4^4A.4.1  Overview 

The  Extended  Air  Defense  Simulation  has  provided  a  valuable  tool  for  operational 
architecture  study  for  many  years.  As  the  understanding  of  possible  architectures  has 
evolved,  so  has  the  flexibility  of  EADSIM  to  represent  the  possible  architectures.  A 
number'of  enhancements  that  directly  relate  to  capabilities  needed  within  current 
concepts  have  been  incorporated  into  EADSIM  via  the  release  of  Version  8.00  and 
most  recently  Version  9.00.  In  addition,  work  is  ongoing  that  directly  supports  the  ability 
to  represent  the  JTAMD  OA  and  the  FoS  that  will  operate  within  that  architecture. 

AvTv2A.4.2  Design 

EADSIM  Version  8.00  made  few  modifications  directly  to  the  areas  identified  above; 
however,  there  were  two  major  enhancements  that  are  key  drivers  to  analysis  in  this 
arena.  The  first  enhancement  was  the  addition  of  the  ISAAC  modules  for  the  ABL  SPO 
approved  representation  of  the  ABL.  These  modules  handle  the  slewing  and  lethality 
computations  for  the  ABL,  while  EADSIM  internal  processing  handles  the  location  of  the 
ABL  and  the  threat,  as  well  as  the  battle  management  for  selecting  engagements  within 
a  single  ABL  and  between  multiple  ABLs  operating  in  the  same  region. 

The  second  related  enhancement  for  Version  8.00  was  the  incorporation  of 
Reliability/Availability/Maintainability  (RAM)  specification  and  modeling  to  allow  detailed 
reliability  statistics  at  the  system  and  element  levels.  This  capability  allows  the  system 
and  components  of  the  system,  such  as  sensors  and  communications  devices  to  be 
disabled  based  on  failure  statistics.  The  disabled  system  or  component  would  become 
unavailable  until  repairs  are  made.  Components  that  are  not  modeled  for  purposes 
other  than  RAM  can  be  represented  down  to  any  level  in  the  typical  work  breakdown 
structure.  A  generator  would  be  an  example  of  such  a  component. 

EADSIM  Version  9.00  introduced  some  significant  enhancements  in  the  area  of  track 
processing.  The  representation  within  EADSIM  without  these  enhancements  provided 
perfect  correlation  of  tracks  for  engagement  related  purposes;  thus,  making  it  difficult  to 
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show  an  improvement  from  the  addition  of  better  correlation  and  tracking  techniques 
that  are  envisioned  with  SIAP  and  CEC.  These  enhancements  also  included  specific 
upgrades  to  IFF  and  Combat  ID  techniques  to  improve  the  representation  of  injection  of 
identification  information  into  the  air  picture.  These  enhancements  allow  a  truer 
representation  of  impacts  of  current  and  envisioned  tracking  systems. 

Track  processing  was  enhanced  in  a  number  of  ways.  The  primary  enhancement  is 
that  a  single  track  file  may  now  contain  multiple  track  entries  on  the  same  object. 
Correlation  of  tracks  is  now  evaluated  on  a  source  by  source  basis.  A  correlation  is  not 
assumed  to  be  valid  for  the  entire  scenario.  Correlations  may  now  be  revisited  multiple 
times  during  a  scenario  run,  based  on  a  user-defined  correlation  revisit  time.  Statistical 
correlation  will  now  miscorrelate  incoming  tracks  into  new  track  entries.  The  user  may 
specify  to  correlate  tracks  probabilistically  and/or  based  on  track  error  volumes. 

Engagement  processing  is  now  performed  based  on  track  number  rather  than  target  ID 
number.  When  a  platform  sends  a  command  message  to  or  receives  a  command 
message  from  another  platform,  the  message  contains  a  track  number  for  the  target.  If 
the  receiving  platform  does  not  recognize  the  track  number,  it  sends  an  update  request 
message  to  the  source  of  the  message.  Upon  receipt  of  the  update  request,  the 
sending  platform  sends  a  commanded  track  update,  which  contains  the  track  data  for 
the  target  as  well  as  the  original  command  data.  The  receiving  platform  processes  the 
track  update  into  its  track  file  and  then  continues  processing  the  original  command 
message.  The  number  of  the  target  track  is  also  stored  in  an  engager’s  target  record 
and  launch  record,  and  processing  of  the  engagement  is  performed  using  this  track 
number.  This  allows  platforms  to  engage  a  target  multiple  times  if  multiple  tracks  are 
held  on  the  same  target  due  to  miscorrelation. 

A  resolution  specification  has  been  added  to  the  IFF  parameters  specified  on  the 
system  definition.  The  resolution  model  allows  any  platform  within  the  resolution  cell  of 
the  interrogated  platform  to  influence  the  outcome  of  the  interrogation.  If  any  platform 
within  the  cell  responds  as  friendly,  the  interrogated  platform  will  be  marked  as  friendly. 
Individual  IFF  modes  may  be  defined  with  different  resolution  parameters  and  a  mode 
list  may  be  specified  for  each  instance  of  performing  IFF.  The  mode  list  indicates  the 
modes  to  be  used  for  performing  the  interrogation,  in  order  of  preference. 


A.5  Extended  Air  Defense  Testbed  (EADTB) 

A.5.1  Overview 

The  Extended  Air  Defense  Testbed  (EADTB)  is  an  event-stepped,  constructive 
simulation  capable  of  real-time,  interactive,  or  batch  mode  operation.  It  was  developed 
for  joint-service,  international  use,  with  the  primary  goal  of  serving  the  extended  air 
defense  community.  Presently  hosted  on  Silicon  Graphics  Onyx/Challenge,  Octane,  or 
Origin  2000  hardware,  the  EADTB  supports  modeling  from  the  battery/fire-unit  level  up 
to  theater-level  scope  with  a  high  degree  of  flexibility  in  choice  of  levels  of  detail  and 
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aggregation.  The  EADTB  incorporates  a  capability  for  detailed,  explicit  simulation  of 
elements  of  C4ISR. 

A.5.2  Design 

Most  of  the  EADTB  software,  as  delivered  to  the  user,  consists  of  object  code, 
precompiled  from  ADA  source  code.  In  addition,  the  EADTB  includes  a  user-modifiable 
(or  definable)  interpreted  ruleset  language  that  invokes  selected  precompiled  EADTB 
algorithms  and  defines  the  simulated  weapon-system  “thinker”  behavior.  The  Thinker 
ruleset  can  also  be  configured  to  access  user-defined,  precompiled  externals.  By 
placing  weapon-system  modeling  power  in  the  hands  of  the  user,  the  EADTB  serves  as 
a  model-development  environment  as  well  as  a  complete  simulation. 

Av§Ai6_General  Campaign  Analysis  Model  Core  Tool  Suite  (GCAM-CTS) 
Ar:k4A.6.1  Overview 

The  GCAM-CTS  was  originally  developed  in  support  of  the  Navy  Assessment  Division 
(OPNAV  N81),  and  is  a  component  of  the  Joint  Analytic  Model  Improvement  Program 
(JAMIP)  near-term  suite  of  models.  GCAM-CTS  is  a  PC-based  set  of  object-oriented 
modeling  and  analysis  tools.  Using  the  GCAM-CTS,  analysts  develop  object-oriented 
stochastic  modeling  applications,  called  GCAM  cases,  tailored  to  particular  issues.  The 
GCAM-CTS  was  certified  as  compliant  with  the  guidelines  of  the  Defense  Modeling  and 
Simulation  Office  (DMSO)  High  Level  Architecture  (HLA)  in  September  1999. 

A4^A.6.2  Design 

Figure  1  shows  a  typical  analysis  and  modeling  process  for  a  notional  study  employing 
GCAM.  One  begins  by  developing  the  two-sided  scenario  and  overarching  context  in 
which  performance  is  to  be  assessed.  The  level  of  detail  and  specific  tactical  features 
incorporated  in  a  GCAM  scenario  are  tailored  to  the  objectives  and  issues  of  the  study. 
GCAM  is  an  integrator,  rather  than  originator,  of  most  mission-level  performance 
analysis.  Thus,  Figure  1  shows  that  external  models  have  been  employed  to  create  a 
database  of  tables  or  performance  curves  in  four  key  mission  areas  .  The  mission  areas 
and  supporting  models  would  depend  on  the  study,  desired  level  of  supporting  detail, 
and  availability  of  analysis  tools  within  the  analysis  team.  A  variety  of  mission 
effectiveness  algorithms  and  campaign  features  can  be  implemented  directly  in  GCAM 
or  interfaces  using  HLA,  Visual  Basic,  Excel,  or  an  external  model.  The  analyst  can 
tailor  GCAM  and  the  supporting  toolset  to  the  scope  and  tempo  of  the  study. 
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Ar4^3A.6.3  Assumptions  and  Limitations 

GCAM  is  limited  only  by  the  level  of  detail  the  user  can  acquire.  The  analyst  decides 
what  level  of  fidelity  to  model  each  system.  He  also  decides  how  these  systems 
interact  with  each  other  and  what  results  to  output.  Inordinate  amounts  of  information, 
though,  may  result  in  excess  run  time,  thus  it  is  best  if  methodologies  are  generalized 
and  assumptions  are  made.  This  is  why,  as  previously  mentioned,  GCAM  is  best  used 
as  an  integrator  of  results. 

At4v4A.6.4  S1AP  Contribution 

GCAM  will  roll  up  results  from  other  lower  level  SIAP  models  described  in  Appendix  A  in 
a  similar  method  as  described  in  Figure  1  above.  For  instance,  it  can  be  used  to  show 
the  impact  mis-correlations  have  on  leakers  over  the  course  of  an  entire  campaign. 
More  detailed  uses  as  they  pertain  to  the  individual  Block  n  analysis  is  contained  in 
subsequent  appendices. 


A.7  Joint  Combat  Identification  Evaluation  Team  (JCIET) 

A.7.1  Overview 

JCIET  is  chartered  to  employ  the  equipment  and  personnel  of  all  Services  to  evaluate, 
investigate,  and  assess  Joint  integration  and  interoperability  of  systems,  concepts, 
capabilities,  Tactics,  Techniques,  and  Procedures  (TTP),  and  doctrine  which  directly 
affect  combat  ID  within  the  present  and  future  Joint  battlespace. 


Page  A-9 
26  April,  2002 


A.7.2  Design 


JCIET  evaluates  3  mission  areas  -  surface-to-surface,  air-to-surface,  and  air 
defense(air-to-air  and  surface-to-air).  The  evaluations  are  done  as  follows: 

•  Utilize  active,  guard,  and  reserve  personnel  with  currently  fielded  equipment. 

•  Coordinate  with  Services,  battle  laboratories,  doctrine  commands,  and  tactics 
schools. 

•  Joint  environment/scenario  for  emerging  technology. 

•  Robust  scenarios  produce  shooter-level  “fog  of  war”  yet  small  enough  to  be 
fully  instrumented. 


A£A.8  J8AAT  The  Joint  Staff  J8  Architectural  Assessment  Tool 


ArT4A.8.1  Overview 

The  J8  Architectural  Assessment  Tool  (J8AAT)  is  an  analytic,  expected  value, 
functional  model  for  estimating  the  interoperability  and  effectiveness  of  an  air  defense 
architecture.  J8AAT  was  originally  developed  under  the  sponsorship  of  J8  to  assist  in 
the  analysis  of  acquisition  alternatives  for  cruise  missile  defense  (CMD)  in  a  joint 
environment  and  was  first  used  in  the  Joint  Land  Attack  Cruise  Missile  Defense  study 
(JLACMD)  to  supplement  force-on-force  models. 

A;4t2A.8.2  Design 

J8AAT  is  a  direct  functional  model  rather  than  a  simulation.  It  uses  a  specified  laydown 
of  blue  force  assets  but  does  not  follow  individual  target  tracks  through  the  theater 
based  on  a  detailed  scenario  (an  exception  to  this  is  made  for  CID  which  requires 
individual  tracks  to  build  up  track  history).  Instead,  at  each  point  of  the  theater,  metrics 
are  computed  for  a  target  at  a  specified  altitude  and  current  direction  of  motion.  The 
orientation  of  the  aircraft  is  primarily  used  to  determine  the  aspect  angle  of  each  radar 
system  with  respect  to  the  target  for  RCS  determination.  The  function  model  design 
requires  generalizations  that  are  not  required  of  a  simulation,  however  these  limitations 
are  offset  by  the  compensating  benefits  of  providing  theater-wide  maps  of  metrics  and 
system  of  systems  capability.  Whereas  simulations  have  to  be  run  many  times  to 
understand  expected  outcomes,  J8AAT  rapidly  produces  global  expected  values.  For 
this  reason,  J8AAT  can  be  a  valuable  augmentation  to  simulations,  since  it  can  quickly 
build  representations  of  new  systems  and  prioritize  major  factors  that  help  in  the  design 
and  execution  of  large  scale  simulations  or  hardware  in  the  loop  experiments. 

ArTr3A.8.3  Assumptions  and  Limitations 
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A4v4A.8.4  SIAP  Contribution 


The  J8AAT  model  is  most  effectively  used  as  a  rapid  turn-around  tool  for  assessing 
proposed  laydowns  and  scenarios  for  exercises  to  see  how  a  collection  of  systems  are 
going  to  interact. 

.  IDA  can  compute  standard  CID  KPPs  (Completeness  and  Accuracy)  for  proposed 
scenarios  and  relate  the  impact  of  SIAP  KPPs  on  the  CID  KPPs. 

.  IDA  can  display  selected  Warfighter  benefits  off  the  SIAP  and  CID  capability. 

•  The  following  types  of  engineering  changes  can  be  represented  in  J8AAT  and  the 
effect  on  SIAP  and  CID  metrics  computed: 

.  Changes  to  current  Platform  specific  correlation  algorithms 
.  Changes  affecting  gridlock  and  sensor  registration 
.  Changes  affecting  latency  of  reporting 

.  Changes  to  the  TADIL  architecture  which  impact  latency  or  connectivity 
.  Introduction  of  sensor  netting  and  different  approaches  to  correlating  fused  data 
with  link  data. 

.  While  the  model  does  not  currently  compute  information  loses  due  to 

incompatibilities  in  CID  lexicon  across  platforms,  given  that  the  loses  can  be 
estimated  by  other  means,  the  impact  on  Warfighter  Benefit  from  these  loses  can  be 
analyzed. 


A^A.9  JCTN  and  JDN  Algorithm  Benchmark 
At4t4A.9.1  Overview 

The  JCTN  and  JDN  Algorithm  Benchmark  Environment  (hereafter  shortened  to 
"Benchmark  Environment"  or  just "  Benchmark")  is  an  event-stepped  computer 
simulation  that  provides  the  functionality  and  infrastructure  to  develop  and  score 
algorithms  against  SIAP  performance  metrics  for  multi-platform,  multi-sensor,  multi¬ 
target  tracking,  as  well  as  for  single  sensor  tracking.  It  was  developed  by  a  collaborative 
team  under  co-sponsorship  of  the  Office  of  Naval  Research  (ONR)  and  the  Ballistic 
Missile  Defense  Organization  (BMDO).  The  current  version  of  the  Benchmark  scores 
performance  within  a  composite  tracking  network  and  for  a  reporting  responsibility  (R2) 
network  having  features  of  Link  16  NPG-7.  The  Benchmark  Environment  also  is 
adaptable  and  expandable  to  deal  with  other  technical  issues  pertinent  to  the  SIAP  SE 
program,  e.g.,  composite  classification  and  identification.  Benchmark  is  intended  to  run 
on  desktop  computers  and  is  coded  in  the  high  level  programming  language  that  is 
provided  as  part  of  the  MATLAB  integrated  technical  computing  environment  developed 
by  The  Math  Works,  Inc.  (www.mathworks.com). 

A4^A.9.2  Design 
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In  its  current  form,  the  Benchmark  Environment  can  evaluate  the  performance  of  test 
article  algorithms  in  the  face  of  difficult,  real-world  tracking  and  track  management 
issues.  These  include  sensor  measurement-to-local  track  data  association,  track 
initiation,  local  track-to-network  track  correlation,  track  promotion  to  composite  track  or 
assumption  of  reporting  responsibility,  track  merge  and  track  drop  logic,  balancing 
performance  across  competing  performance  evaluation  metrics  (e.g.,  completeness 
versus  redundancy),  navigation  errors,  coordinate  frame  transformations  and  their 
impact  on  error  estimates,  gridlock,  data  latency  and  dropped  messages,  and  others. 

The  Benchmark  Environment  includes  high  fidelity,  operationally  realistic  scenarios  that 
present  difficult  tracking  situations  including  merging  and  crossing  air  traffic,  close 
formations  of  maneuvering  aircraft,  high-g  breaks,  low-altitude/low-observable  targets, 
terrain  masking,  etc.  It  can  easily  accommodate  new  and  varied  scenarios,  such  as 
JCIET  exercise  scenarios. 

In  terms  of  tracking,  data  association,  and  track  correlation,  the  JCTN  and  JDN  are 
intended  to  provide  the  capability  for  platforms  at  different  locations  to  maintain  a  SIAP. 
Test  article  algorithm  performance  is  evaluated,  in  part,  based  on  the  consistency  of  the 
track  databases  maintained  at  different  nodes  in  the  network.  The  Benchmark 
Environment  simulates  the  track  databases  at  each  node/platform  as  separate  entities. 

A4^3A.9.3  Assumptions  and  Limitations 

AA4A.9.4  SiAP  Contribution 

One  of  the  key  elements  of  the  Benchmark  Environment  is  the  metrics  module.  The 
Benchmark  metrics  are  based  on  evaluating  the  following  SIAP  attributes  which  were 
originally  formulated  by  the  SIAP  splinter  groups  of  the  1999  Joint  Mission  Area 
Assessment  (JMAA)  for  Countering  Theater  Air  and  Missile  Threats.  The  performance 
metrics  currently  implemented  in  the  Benchmark  include: 

.  Completeness,  specifically  (1 )  object  track  completeness 
.  Timeliness,  specifically  (2)  track  initiation  time 

•  Track  continuity,  including  (3)  mean  cumulative  switches  of  tracks  and  (4)  mean 
cumulative  breaks  of  tracks 

.  Ambiguity,  including  (5)  redundant  track  mean  ratio  and  (6)  spurious  track  mean 
ratio 

.  Accuracy,  including  (7)  track  accuracy  and  (8)  track  covariance  consistency 
.  Cross-platform  commonality  history,  including  (9)  ratio  of  non-common  track 
numbers  and  (10)  track  state  estimate  differences 

The  Benchmark  also  scores  (11)  the  total  source  message  data  load  presented  to  the 
communications  network. 
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A.10  Joint  Distributed  Engineering  Plant  Track  2  (JDEP  Track  2) 

A.10.1  Overview 

The  JDEP  program  was  established  as  a  DOD-wide  effort  to  link  existing  service  and 
joint  combat  system  engineering  and  test  sites  (including  design  activities,  software 
support  activities,  test  and  evaluation  facilities,  training  commands,  and  operational 
units).  JDEP  is  designed  to  improve  the  interoperability  of  weapon  systems  and 
platforms  through  rigorous  testing  and  evaluation  in  a  replicated  battlefield  environment. 


ArSA.11  Joint  Interim  Mission  Model  (JIMM) 

AT4A.11.1  Overview 

The  Joint  Interim  Mission  Model  (JIMM)  is  being  developed  by  the  Joint  Strike  Fighter 
Program  Office  (JSF  PO)  to  cover  the  acknowledged  gap  in  next-generation  modeling 
at  the  mission  level  for  analysis.  JIMM  may  be  used  for  both  constructive  (digital  only) 
or  virtual  (real-time,  hardware-  or  operator-in-the-loop)  simulation. 

JIMM  is  the  merger  of  two  legacy  simulations:  SWEG  and  Suppressor.  Each  of  these 
two  simulations  has  a  rich  heritage  of  virtual  and  constructive  applications,  with  SWEG 
better  known  for  its  virtual  capabilities,  while  Suppressor  is  known  for  constructive 
analysis.  By  merging  these  two  simulations  into  a  common,  object-oriented  simulation 
written  in  C++,  JIMM  will  leverage  the  VV&A  history  of  both  legacy  tools,  while  lowering 
the  maintenance  costs  and  increasing  the  user  base. 

A4t2A.11.2  Design 

JIMM  models  real-world  entities  via  user-defined  functional  objects.  Since  no 
preconceived  notions  of  what  constitutes  an  aircraft,  ship,  or  tank  are  in  the  software, 
JIMM  has  inherent  flexibility  to  a  wide  range  of  applications.  The  functional  objects 
include: 

.  Sensors,  for  the  non-cooperative  exchange  of  information  [radars,  human  eye] 

.  Communicators,  for  the  cooperative  exchange  of  information  [radios,  land  line] 

.  Weapons,  for  lethal  engagement  of  others  [bombs,  guns] 

.  Disruptors,  for  non-lethal  engagement  of  others  [jammers,  chaff] 

.  Movers,  for  changing  location  over  time  [truck  chassis,  aircraft  engine] 

.  Thinkers,  for  processing  of  data  and  decision  making  [targeting  cells,  computer  chip] 

Other  objects  also  exist,  such  as  Tactics  for  rules  of  behavior,  Groups  for  message 
definition,  Elements  for  susceptibility  and  vulnerability  modeling,  Shapes  for  physical 
extent,  and  Resources  for  representing  expendables  and  consumables. 
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JIMM  supports  real-time  operation  as  an  inherent  part  of  its  design.  A  user-definable 
interface  partitions  the  functional  objects  within  a  scenario  to  JIMM  or  other  assets, 
depending  upon  the  needs  of  the  analyst.  In  addition  to  a  robust  set  of  internal 
protocols  and  data  definitions,  JIMM  also  has  interfaces  for  standard  distributed 
simulation  protocols  and  architectures,  including  DIS  and  HLA.  These  interfaces  allow 
JIMM  to  be  integrated  with  other  simulations  or  simulators,  such  as  ESAMS  or  FRED 
via  Shared  Memory.  These  same  interfaces  are  also  used  to  integrate  JIMM  with 
hardware  systems  for  more  detailed  simulation  and  stimulation  applications. 

Because  of  the  data-driven  nature  of  JIMM,  any  discussion  would  be  incomplete  without 
mentioning  databases.  Legacy  SWEG  databases  from  version  6.5.0  and  beyond  are 
directly  readable  by  JIMM.  Suppressor  has  a  robust  set  of  databases  that  have  been 
developed  and  refined  over  many  years.  Special  effort  has  been  made  to  ensure  these 
modeled  systems  are  translatable  into  JIMM.  To  achieve  this,  a  Database  Converter 
has  been  developed  to  automatically  convert  most  of  an  existing  Suppressor  database 
into  JIMM  format.  Those  data  items  not  converted  are  flagged  for  analyst  action  or  as 
being  not  currently  implemented  in  JIMM.  Plans  are  in  place  to  eliminate  this  latter 
category. 

Finally,  standard  analyst  tools  and  interface  libraries  are  being  developed  for  JIMM. 
These  include: 

.  Software  for  exchanging  data  through  JIMM’s  Shared  Memory  structures 
.  Conversion  of  standard  intelligence  community  data  sets  into  JIMM  format 
.  Simulation  control  software  and  run  scripts 
.  Data  extraction,  formatting,  and  display  tools 
.  Graphical  displays  for  real-time  monitoring  and  post-processing 
*  Extraction  and  display  of  detailed  JIMM  data  items 

.  Pre-processing  and  run-time  Graphical  User  Interfaces  (GUIs) 


A^A.12  Network  Design  Analysis 
A4t4A.12.1  Overview 

The  systems  planned  to  be  used  for  the  distribution  of  SIAP  information  are  usually  also 
intended  to  distribute  other  types  of  information  as  well  (e.g.,  commands,  etc.).  This 
means  that  the  amount  of  capacity  allocated  to  the  SIAP  functions  must  compete  with 
those  other  functions.  There  are  many  primary  SIAP  attributes  that  are  affected  either 
directly  or  indirectly  by  the  amount  of  capacity  allocated,  and  the  degree  of  connectivity 
provided,  to  various  SIAP  and  non-SlAP  functions  during  the  network  design  process. 

In  addition,  those  other  functions  also  provide  warfighting  benefits.  Therefore,  it  is 
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necessary  to  develop  a  balance  between  allocation  of  capacity  to  the  SIAP  and  non- 
SIAP  functions  within  the  theater  network  architecture. 

Before  any  SIAP  attributes  can  be  assessed,  it  is  necessary  to  make  some  top  level 
trades  among  the  various  SIAP  and  non-SlAP  functions.  A  network  design  analysis  is 
required  to  ensure  that  changes  to  the  network  architecture  or  data  flows  proposed  in 
the  name  of  SIAP  do  not  inadvertently  impact  some  of  the  critical  non-SlAP  functions. 
This  is  required,  since  it  is  possible  that  any  warfighting  benefits  accrued  by  using  more 
capacity  to  improve  a  specific  SIAP  attribute  may  be  either  partially  or  totally  offset  by 
degradation  in  another  non-SlAP  function. 

ArT^A.12.2  Design 

The  Services  have  developed  a  Joint  Link  16  Network  Design  Aid  (JNDA),  which 
provides  a  mechanism  for  designing  networks  to  support  the  full  range  of  SIAP  and 
non-SlAP  functionality  that  needs  to  be  supported  by  Link  16  in  a  theater  network 
architecture.  Using  Service  network  design  guidelines,  a  baseline  theater  network 
architecture  will  be  created,  primarily  focused  on  Link  16,  though  the  baseline  Link  16 
network  could  also  incorporate  capacity  and  features  needed  to  port  data  from  other 
theater  networks  (e.g.,  CEC,  JCTN,  or  other  TADILs)  into  Link  16  as  well. 

When  SIAP  related  changes  are  proposed  which  require  changes  in  the  capacity 
allocations  of  the  baseline  network,  the  impact  of  the  shift  in  capacity  will  be 
determined,  and  expressed  in  terms  of  the  appropriate  MOEs  and  MOPs.  The  network 
design  analysis  will  attempt  to  express  the  gains  and  losses  in  SIAP  and  non-SlAP 
performance  and  functionality  in  terms  suitable  for  inclusion  in  the  Decision  Support 
Binders,  which  will  accompany  recommendations  to  the  JROC  regarding 
implementation  of  various  SIAP  related  changes. 


/W1QA.13  Operational  Data  Driven  Simulation  for  Correlation  Algorithm 
Performance  Evaluation  (ODDSCAPE) 

ArTTA.13.1  Overview 

The  ODDSCAPE  modeling  and  simulation  tool  was  developed  at  CNA  to  provide 
predictive  analysis  of  proposed  Link-16  correlation  /  decorrelation  algorithms. 
ODDSCAPE  builds  a  local  air  picture  for  each  participating  unit  from  recorded  combat 
system  air  picture  data.  Link  communications  are  emulated  and  rule  sets  are 
implemented  to  produce  track  reporting  over  TADIL-J.  Then  each  unit  uses  a  set  of 
proposed  correlation  /  decorrelation  rules  (such  as  the  Corr  /  Decorr  ICP)  to  merge  the 
unit’s  local  air  picture  to  the  remote  reports  incoming  over  the  link. 

Ai4v2A.13.2  Design 
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We  drive  the  simulation  primarily  by  the  local-sensor-support  data  recorded  on  the 
participating  units  during  ASCIETOO.  Local  sensor  updates  are  delivered  to  units  at  the 
appropriate  time  step.  For  remote  track  updates,  a  unit  in  the  simulation  receives  J3.2 
track  update  messages  on  tracks  reported  by  other  units,  applies  the  appropriate  logic 
(including  correlation  /  decorrelation  tests),  and  updates  its  track  database  as 
necessary.  Each  unit  also  receives  the  identical  J2.2  PPLI  messages  it  received 
during  ASCIETOO.  Correspondingly,  any  local-sensor-supported  tracks  that  are  not 
correlated  to  incoming  remote  reports  are  reported  on  the  link  (in  accordance  with  the 
MIL  STD  and  the  ICP)  and  subsequently  received  (after  some  delay)  by  the  other 
participating  units.  Finally,  we  attempt  to  reproduce  the  observed  operator-initiated 
messages  that  may  affect  correlation  /  decorrelation  decisions  by  feeding  “injection 
prompts”  to  a  unit.  A  unit  receiving  an  injection  prompt  will  create  and  send  the 
specified  message  in  the  simulation  that,  hopefully,  replicates  the  message  observed  in 
the  ASCIET  00  data. 


A44A.14  Performance  Evaluation  Tool  (PET) 

ArT4A.14.1  Overview 

PET  is  a  PC-based  computer  program  that  reads  many  data  types  and  compares 
events  automatically  to  calculate  interoperability  metrics.  PET  also  displays  metrics 
graphically  over  time  and  allows  analysts  to  step  through  the  metrics  chronologically  to 
assist  in  tying  metrics  to  performance  issues.  PET  was  developed  to  load  combat 
system  data  and  calculate  metrics  within  the  seven  levels  that  related  primarily  to 
tracking  and  identifying  aircraft,  beginning  with  “connectivity”  and  ending  with 
“Battleforce  situational  awareness”. 

Arlr£A.14.2  Design 

Normal  PET  usage  starts  with  loading  data  from  all  combat  systems  to  be  analyzed, 
reconstructing  the  tracks  that  represent  aircraft  and  surface  units  of  interest,  setting 
independent  variables  for  how  the  data  is  to  be  interpreted,  and  producing 
interoperability  measures  results  and  interpretations.  This  subsequent  interpretation  of 
metric  results  is  done  by  producing  graphical  displays  of  metrics  over  time  and  tying 
significant  deviations  in  metric  numbers  to  combat  system  events,  and  by  detailed 
combat  systems  analysis.  When  the  data  is  reconstructed  and  calculation  rules 
characterized,  the  analyst  can  calculate  the  metrics  and  output  tabular  results  and 
graphical  representations  of  the  analysis. 

PET  is  currently  programmed  to  calculate  three  sets  of  metrics.  The  CNO  801  metrics, 
the  ASN  RDA  CEC  metrics  and  the  Timeframe  metrics  used  by  JCIET  for  JCIET 
evaluations  and  CNA  and  NWAS  to  analyze  a  variety  of  other  joint  events.  The  ASN 
RDA  metrics  will  be  used  throughout  the  CEC  testing  and  may  be  further  applied  to 
events  where  the  primary  focus  is  characterization  of  what  is  in  various  track  files.  The 
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Timeframe  metric  set  has  been  gaining  increasing  acceptance  since  the  measures 
describe  what  happens  over  time  and  are  directly  related  to  what  an  operator  sees.  A 
track  file  metric  is  generally  used  to  characterize  how  one  track  file  compares  to  another 
(i.e.  how  does  the  combat  system  track  file  with  CEC  on  compare  with  the  track  file  with 
CEC  off)?  A  timeframe  metric  is  generally  used  to  describe  what  the  operator  is  seeing 

overtime. 


A.15  Roving  Sands  (RS) 

A.  15.1  Overview 

Roving  Sands  (RS-01)  is  a  one-of-a-kind  event  that  is  the  world's  largest  joint  theater  air 
and  missile  defense  exercise.  RS-01  melds  the  command,  control,  communications  and 
computer  elements,  air  defense  artillery  and  aircraft  of  the  Army,  Navy,  Marines,  Air 
Force  and  multinational  forces  into  a  joint  integrated  air  defense  system.  Roving  Sands 
01  is  one  of  a  series  of  training  opportunities  that  provides  deployable  forces  with  an 
enhanced  understanding  of  joint  and  multinational  operations  and  tasks. 

Roving  Sands  will  be  conducted  at  training  ranges  and  sites  in  Texas  and  New  Mexico. 
The  training  objectives  for  this  exercise  reflect  a  wide  range  of  capabilities  that  may  be 
needed  in  various  geographic  areas.  This  training  will  enhance  the  ability  of 
commanders  and  staffs  to  plan  and  conduct  joint  and  combined  tactical  air  operations 
and  theater  missile  defense  operations  under  realistic  conditions. 

During  this  exercise,  the  forces  will  refine  their  inter-operability  skills  using  a  joint  and 
combined  intergrated  air  defense  network  of  ground,  missile  and  radar  early  warning 
systems.  They  will  face  an  opposing  force  of  tactical  aircraft,  ballistic  and  cruise 
missiles  in  a  high-threat  environment.  To  do  this,  Army,  Marine  and  a  contingent  of 
multinational  air  forces  will  employ  air  defense  systems,  such  as  the  Patriot  anti-tactical 
missile  system,  against  realistic  front-line  attack  forces  provided  by  the  U.S.  Air  Force. 


A.1 6  Theatre  Missile  Defense  System  Exerciser  (TDMSE) 

A.  16.1  Overview 

The  TMDSE  is  a  joint  services  emulation  of  theater  missile  defense  using  Link-16 
emulation  and  hardware-in-the-loop  to  integrate  the  theater  missile  defense  family  of 
systems  and  test  interoperability  issues  between  the  separately  developed  systems. 
Connectivity  is  accomplished  by  Data  Link  Gateway  systems  at  each  participating  site 
networked  over  classified  T1  lines,  thus  enabling  a  virtual  Link-1 6  network.  TMDSE  is 
sponsored  by  the  Ballistic  Missile  Defense  Organization. 
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Ar4£A.17  Task  Force  Exerciser  (TFX) 


A4-4A.17.1  Overview 

The  Task  Force  Exerciser  (TFX)  provides  a  capability  to  realistically  stimulate  and 
stress  U.S  Army  Theater  Air  and  Missile  Defense  (TAMD)  systems  operating  in  a  task 
force  configuration.  It  leverages  past  and  current  development  efforts  expended  for  the 
Theater  Missile  Defense  System  Exerciser  (TMDSE)  including  the  Test  Exercise 
Controller  (TEC),  appropriate  tactical  drivers,  and  planned  upgrades.  This  approach 
minimizes  new  development  and  concentrates  on  incorporating  U.S.  Army  Air  and 
Missile  Defense  (AMD)  elements  and  uniquely  required  communications.  The  TFX 
provides  a  cost-effective  tool  to  exercise  and  test  the  integration  and  interoperability  of 
multiple  U.S.  Army  AMD  tactical  battle  management  command,  control,  and 
communications  (BMC3)  systems  hardware  and  software.  It  retains  compatibility  with 
TMDSE  to  leverage  past  and  future  Ballistic  Missile  Defense  Organization  (BMDO) 
investments  in  the  use  of  distributed  interactive  simulation  technology  to  test-analyze- 
fix-test  AMD  systems. 

A^A.17.2  Design 

Doctrinally,  the  TAMD  Task  Force  is  an  evolving  concept  that  envisions  a  two-tiered 
(THAAD  and  PATRIOT)  defense  against  tactical  ballistic  missiles  (TBMs)  and  air- 
breathing  threats  (ABTs).  SHORAD  systems  have  been  integrated  into  the  Task  Force 
to  provide  additional  capabilities  against  ABTs,  to  include  cruise  missiles  (CMs)  and 
unmanned  aerial  vehicles  (UAVs). 

TFX  provides  a  framework  for  demonstrating  interoperability  between  tactical  systems 
and  exploring  developmental  and  operational  interoperability  issues  by  stimulating 
participant  systems  and  measuring  their  responses. 


A.18  Virtual  Warfare  Center 
A.18.1  Overview 

A  collaborative,  immersive  development  environment  for  war  fighters  and  commanders 
(CINC,  JFACC,  etc).  Encompasses  over  15,000  square  feet  for  JCMD,  computers, 
networks  and  visual  equipment. 


A43A.19  Warfare  Assessment  Model  (WAM) 
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A4t4A.19.1  Overview 


JCIET,  NWAS,  AWACS,  E-2C  and  CNA  currently  use  WAM  as  a  display  tool  to  support 
analysis.  Data  displayed  include  ASCII  files  created  from  AEGIS,  Patriot,  TAOM,  E-2C, 
AWACS,  ABMOC,  AOC,  Rivet  Joint,  Senior  Scout,  EP-3,  TSPI  and  JTIDS  terminal 
extract  (track  and  link  data). 

A?4t2A.19.2  Design 

One  advantage  of  WAM  is  its  capability  to  load  track  data  from  the  above  systems  and 
color-code  it  by  source.  Subsequently,  tracks  can  be  compared  by  displaying  them  "on- 
top"  of  each  other.  Tracking  deviations  then  stand  out.  WAM  also  has  the  capability  to 
display  engagements,  ESM  lines  of  bearing,  and  a  brief  summary  of  track  parameters 
such  as  R2,  ID  parameters,  IFF  Modes  and  geographic  information. 


A.20  Wargame  2000  (WG2K) 

A.20.1  Overview 

The  Wargame  2000  System  is  the  successor  to  the  Advanced  Real-time  Gaming 
Universal  Simulation  (ARGUS),  which  is  a  real-time,  interactive,  discrete  event, 
command  and  control  missile  defense  simulation.  Wargame  2000  is  intended  to  provide 
a  simulated  combat  environment  that  will  allow  war-fighting  commanders,  their  staffs, 
and  the  acquisition  community  to  examine  missile  and  air  defense  concepts  of 
operation  (CONOPS).  CONOPS  includes  doctrine,  tactics,  techniques,  and  procedures. 
CONOPS  is  an  integral  part  of  larger  combat  environment.  WG2000  will  support 
CONOPS  evaluation  through  the  use  of  human-in-control  experiments  and  other 
events.  The  Wargame  2000  System  is  intended  to  provide  a  robust,  flexible,  easy-to- 
use  architecture,  which  incorporates  current  as  well  as  accommodates  evolving  weapon 
characteristics  to  conduct  missile  and  air  defense  investigations 
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